Extra none:none images after docker build I am quite new on docker and tried to build a postgres database image with preset database scheme created on. Thus created below dockerfile

#alpine linux version of the postgres image. uses a really small amount of disk
FROM postgres:alpine

#environment database name that postgres will get to create during the startup
ENV POSTGRES_DB=test

#Superuser name that will be created during the container creation
ENV POSTGRES_USER=test

#superuser password to be used to connect to the postgres instance
ENV POSTGRES_PASSWORD=test

#sql file to be run while container getting created
COPY auto-create-scheme.sql /docker-entrypoint-initdb.d/

but once the modified image created. I got below output;

docker build --no-cache -t test .
  Sending build context to Docker daemon  28.67kB
  Step 1/5 : FROM postgres:alpine
   ---> 47510d0ec468
  Step 2/5 : ENV POSTGRES_DB=test
   ---> Running in 7d0246b1cb20
  Removing intermediate container 7d0246b1cb20
   ---> 8be2b9662e86
  Step 3/5 : ENV POSTGRES_USER=test
   ---> Running in 51982d642f99
  Removing intermediate container 51982d642f99
   ---> 954931dc66a8
  Step 4/5 : ENV POSTGRES_PASSWORD=test
   ---> Running in a0ce506ccf78
  Removing intermediate container a0ce506ccf78
   ---> c97298682dac
  Step 5/5 : COPY create-scheme.sql /docker-entrypoint-initdb.d/
   ---> 95df9ff93fcd
  Successfully built 95df9ff93fcd
  Successfully tagged test:latest

following this; if I run docker images -a

I can see those image layers as repo and tag values none.

How can I create my build file to be able discard these image layers have only the final image test.

<none>                           <none>              c97298682dac        5 minutes ago       71.6MB
<none>                           <none>              8be2b9662e86        5 minutes ago       71.6MB
<none>                           <none>              954931dc66a8        5 minutes ago       71.6MB
test                             latest              95df9ff93fcd        5 minutes ago       71.7MB

Those images are the docker build cache. You need this to be able to quickly rebuild similar images, and to avoid rebuilding unchanged layers which would result in resending those to remote registry servers and forcing servers to download those unchanged layers again. One important point on these layers:

They do not consume any significant disk space

You can view the layers of an image by running the docker image history command on an image. Here's an example of a rather large image built on my machine:

$ docker image history 25f90c8ba1ca
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
25f90c8ba1ca        11 days ago         /bin/sh -c #(nop)  CMD ["/bin/sh" "-c" "echo…   0B
8a190467427e        11 days ago         /bin/sh -c #(nop) COPY dir:36bb9e67a0f2844c2…   298MB
1f6f5474071d        11 days ago         /bin/sh -c #(nop) COPY dir:7c364877951bb2312…   249MB
007510a95d17        11 days ago         /bin/sh -c #(nop) COPY dir:9817b6e3d212385d0…   231MB
ee23b7a42488        11 days ago         /bin/sh -c #(nop) COPY dir:57e817f3d9d977f66…   277MB
e190d1bb3f66        11 days ago         /bin/sh -c #(nop) COPY dir:9ba997e629e74ca64…   205MB
abe0ec5c0ef9        11 days ago         /bin/sh -c #(nop) COPY dir:4dbbd5b2010590e28…   213MB
544af937d8a3        11 days ago         /bin/sh -c #(nop) COPY dir:0ac40edba2c7e1317…   178MB
b78085aba431        11 days ago         /bin/sh -c #(nop) COPY dir:2137a075c552d6783…   135MB
3ff8488b067b        11 days ago         /bin/sh -c #(nop) COPY dir:45da2005488e81e9c…   112MB
19b7cdd9d682        11 days ago         /bin/sh -c #(nop) COPY dir:2b745a4311740ddf7…   65.5MB
eb2cf84859fb        11 days ago         /bin/sh -c #(nop) COPY dir:cd7b639f2eee0e9da…   50.1MB
1ba1c21bb0a3        11 days ago         /bin/sh -c #(nop) COPY dir:594739607d1f2d593…   51.9MB
1da5d6bf03e9        11 days ago         /bin/sh -c #(nop) COPY dir:757daf68a395cfaae…   30MB
f6df8b695822        11 days ago         /bin/sh -c #(nop) COPY dir:449b6c2ff93606741…   18.5MB
72895d213dac        11 days ago         /bin/sh -c #(nop) COPY dir:2290eb8d11f15dbe6…   13.8MB
3b6ceadce77b        11 days ago         /bin/sh -c #(nop) COPY file:fcd71a6bfc87e9dd…   167B
52dd0e701be3        11 days ago         /bin/sh -c #(nop)  ENV TARGET=/usr/share/ngi…   0B
aae476eee77d        3 weeks ago         /bin/sh -c #(nop)  CMD ["nginx" "-g" "daemon…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  STOPSIGNAL [SIGTERM]         0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  EXPOSE 80/tcp                0B
<missing>           3 weeks ago         /bin/sh -c #(nop) COPY file:1d1ac3b9a14c94a7…   1.09kB
<missing>           3 weeks ago         /bin/sh -c #(nop) COPY file:af94db45bb7e4b8f…   643B
<missing>           3 weeks ago         /bin/sh -c GPG_KEYS=B0F4253373F8F6F510D42178…   13.3MB
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV NGINX_VERSION=1.15.5     0B
<missing>           4 weeks ago         /bin/sh -c #(nop)  LABEL maintainer=NGINX Do…   0B
<missing>           6 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>           6 weeks ago         /bin/sh -c #(nop) ADD file:25c10b1d1b41d46a1…   4.41MB

$ docker image inspect --format '{{json .RootFS.Layers}}' 25f90c8ba1ca | jq
.
[
  "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
  "sha256:39e8483b9882aa8e95fb00c14e1c5dbdf32e836af4aff4bf862608e0a35bc914",
  "sha256:431a5c7929dd5848b9734d84d5cc5be237dd84e9e5ab1cb84ad6975bdc171c0c",
  "sha256:a83dbde6ba05e8112bd2732375ffbd350d9872d9c9e7e088ba5127118cd6d99e",
  "sha256:65ec7af97662fd3f933dc8b0aefb9f2d9d25c0d78530142bcdab1323c6aee223",
  "sha256:25212046c4c3a7589e45b426085da35d7e5e5a7a834180977a43df533672c360",
  "sha256:508c1d799f8da8accdd8b4e92964c73d6f9b8c4e7381fd0e1241e894d01418d3",
  "sha256:a197e87da4cd4587c8965d81878fbb1d898c8ba7d2a05ff0e9862eb700d26053",
  "sha256:f66588501183b7000c7683808ec4653bb83dc3c07f77d67fc34c78e9d9e5cb33",
  "sha256:de2288e4aa944daa6059184813c549d994f5c2407bfae61c7f8b4e0f950e73b1",
  "sha256:77ae40a90a8d41d53751ee800d4008d7a3706ecae6364560b4606ba0167c2674",
  "sha256:1a4bae64e86919b1f40e61a6022e8f910748566b88c2628893c4e424a60c74a2",
  "sha256:ca61f354107fc14de3c01f5bb192ff8b673b6f84c7089a23d754b23a65751310",
  "sha256:d0da652f2c59d6fb6380c0b6940aaa9f5e5a4fc28ae0b82f1acb5ccf9a1884e1",
  "sha256:5f3a91149be048f4f0a4d2450eab6ff8e630bb416b5c508633014e645f57df06",
  "sha256:0c0762559bcfec46462749615ee06b0998e61673e8118cd1a43e20b38e27825a",
  "sha256:581ad1603f59463febb540052046bbdec82bdfc7c75fd4a94578bdca88b91fec",
  "sha256:ff599edf8583ba1dfd7d9c79c1b718f94641e1fb644a0be29ca5b45dcc1768b4",
  "sha256:6ce4a45f89645a2020cfdae51c83470d9f24424ea16f4bddaf1244aa08f2945e",
  "sha256:5fcfba1ba609a8fbfc3eccbc805eb18a78041897ec3f6dc168c9ed871baed950"
]

$ docker image inspect --format '{{json .RootFS.Layers}}' 8a190467427e | jq .
[
  "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
  "sha256:39e8483b9882aa8e95fb00c14e1c5dbdf32e836af4aff4bf862608e0a35bc914",
  "sha256:431a5c7929dd5848b9734d84d5cc5be237dd84e9e5ab1cb84ad6975bdc171c0c",
  "sha256:a83dbde6ba05e8112bd2732375ffbd350d9872d9c9e7e088ba5127118cd6d99e",
  "sha256:65ec7af97662fd3f933dc8b0aefb9f2d9d25c0d78530142bcdab1323c6aee223",
  "sha256:25212046c4c3a7589e45b426085da35d7e5e5a7a834180977a43df533672c360",
  "sha256:508c1d799f8da8accdd8b4e92964c73d6f9b8c4e7381fd0e1241e894d01418d3",
  "sha256:a197e87da4cd4587c8965d81878fbb1d898c8ba7d2a05ff0e9862eb700d26053",
  "sha256:f66588501183b7000c7683808ec4653bb83dc3c07f77d67fc34c78e9d9e5cb33",
  "sha256:de2288e4aa944daa6059184813c549d994f5c2407bfae61c7f8b4e0f950e73b1",
  "sha256:77ae40a90a8d41d53751ee800d4008d7a3706ecae6364560b4606ba0167c2674",
  "sha256:1a4bae64e86919b1f40e61a6022e8f910748566b88c2628893c4e424a60c74a2",
  "sha256:ca61f354107fc14de3c01f5bb192ff8b673b6f84c7089a23d754b23a65751310",
  "sha256:d0da652f2c59d6fb6380c0b6940aaa9f5e5a4fc28ae0b82f1acb5ccf9a1884e1",
  "sha256:5f3a91149be048f4f0a4d2450eab6ff8e630bb416b5c508633014e645f57df06",
  "sha256:0c0762559bcfec46462749615ee06b0998e61673e8118cd1a43e20b38e27825a",
  "sha256:581ad1603f59463febb540052046bbdec82bdfc7c75fd4a94578bdca88b91fec",
  "sha256:ff599edf8583ba1dfd7d9c79c1b718f94641e1fb644a0be29ca5b45dcc1768b4",
  "sha256:6ce4a45f89645a2020cfdae51c83470d9f24424ea16f4bddaf1244aa08f2945e",
  "sha256:5fcfba1ba609a8fbfc3eccbc805eb18a78041897ec3f6dc168c9ed871baed950"
]

$ docker image inspect --format '{{json .RootFS.Layers}}' 1f6f5474071d | jq .
[
  "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
  "sha256:39e8483b9882aa8e95fb00c14e1c5dbdf32e836af4aff4bf862608e0a35bc914",
  "sha256:431a5c7929dd5848b9734d84d5cc5be237dd84e9e5ab1cb84ad6975bdc171c0c",
  "sha256:a83dbde6ba05e8112bd2732375ffbd350d9872d9c9e7e088ba5127118cd6d99e",
  "sha256:65ec7af97662fd3f933dc8b0aefb9f2d9d25c0d78530142bcdab1323c6aee223",
  "sha256:25212046c4c3a7589e45b426085da35d7e5e5a7a834180977a43df533672c360",
  "sha256:508c1d799f8da8accdd8b4e92964c73d6f9b8c4e7381fd0e1241e894d01418d3",
  "sha256:a197e87da4cd4587c8965d81878fbb1d898c8ba7d2a05ff0e9862eb700d26053",
  "sha256:f66588501183b7000c7683808ec4653bb83dc3c07f77d67fc34c78e9d9e5cb33",
  "sha256:de2288e4aa944daa6059184813c549d994f5c2407bfae61c7f8b4e0f950e73b1",
  "sha256:77ae40a90a8d41d53751ee800d4008d7a3706ecae6364560b4606ba0167c2674",
  "sha256:1a4bae64e86919b1f40e61a6022e8f910748566b88c2628893c4e424a60c74a2",
  "sha256:ca61f354107fc14de3c01f5bb192ff8b673b6f84c7089a23d754b23a65751310",
  "sha256:d0da652f2c59d6fb6380c0b6940aaa9f5e5a4fc28ae0b82f1acb5ccf9a1884e1",
  "sha256:5f3a91149be048f4f0a4d2450eab6ff8e630bb416b5c508633014e645f57df06",
  "sha256:0c0762559bcfec46462749615ee06b0998e61673e8118cd1a43e20b38e27825a",
  "sha256:581ad1603f59463febb540052046bbdec82bdfc7c75fd4a94578bdca88b91fec",
  "sha256:ff599edf8583ba1dfd7d9c79c1b718f94641e1fb644a0be29ca5b45dcc1768b4",
  "sha256:6ce4a45f89645a2020cfdae51c83470d9f24424ea16f4bddaf1244aa08f2945e"
]

In each of those inspect commands you should see that each new image is just adding on an extra layer until you reach the final image. These layers are the same sha256 checksums, pointing to the same files on the disk. Docker does not copy these files per image, it doesn't even copy them when creating containers (excluding initializing named volumes), the layers are read-only and reused by every descendant image and container via a union filesystem. The disk usage that docker displays is the sum of all of the layers even when other images may share layers, while your actual disk usage is only of the unique layers.

If you rebuild images often on the server with the same build tag, you can prune dangling images with docker image prune. Otherwise, simply avoid displaying these build artifacts by running docker image ls without the -a option.


What is the equivalent of eksctl option "--alb-ingress-access" in a config file? I read How to run Serverless Kubernetes: AWS EKS on Fargate, and am wondering what the equivalent option for --alb-ingress-access is in a config schema shown in Config file schema? At least it's not obvious to me. I even created a sample cluster and printed out the info and didn't see it there.
$ eksctl create cluster \
    --name ingress-sample \
    --version 1.19 \
    --region us-east-1 \
    --fargate \
    --alb-ingress-access
$ eksctl get cluster --name ingress-sample -o yaml
- Arn: arn:aws:eks:us-east-1:0123456789:cluster/ingress-sample
  CertificateAuthority:
    Data: <redacted>
  ClientRequestToken: null
  CreatedAt: "2021-02-19T00:44:27.976Z"
  EncryptionConfig: null
  Endpoint: https://<redacted>.gr7.us-east-1.eks.amazonaws.com
  Identity:
    Oidc:
      Issuer: https://oidc.eks.us-east-1.amazonaws.com/id/redacted
  KubernetesNetworkConfig:
    ServiceIpv4Cidr: 10.100.0.0/16
  Logging:
    ClusterLogging:
    - Enabled: false
      Types:
      - api
      - audit
      - authenticator
      - controllerManager
      - scheduler
  Name: ingress-sample
  PlatformVersion: eks.1
  ResourcesVpcConfig:
    ClusterSecurityGroupId: sg-xxxxxx
    EndpointPrivateAccess: false
    EndpointPublicAccess: true
    PublicAccessCidrs:
    - 0.0.0.0/0
    SecurityGroupIds:
    - sg-xxxxxx
    SubnetIds:
    - subnet-111111
    - subnet-222222
    - subnet-333333
    - subnet-444444
    VpcId: vpc-xxxxxx
  RoleArn: arn:aws:iam::0123456789:role/eksctl-ingress-sample-cluster-ServiceRole-D0X7QCIRBSDN
  Status: ACTIVE
  Tags: {}
  Version: "1.19"

UPDATE: Hmmm, maybe
$ eksctl create cluster --alb-ingress-access -f config.yaml

Yep this worked, or at least I didn't get an error
    $ eksctl create cluster --alb-ingress-access -f config.yaml


Nexus vs. Artifactory I was wondering what is the recommendation for a business that needs to store binaries for distribution as well as for dependencies to be used by our projects, with projects in different languages/technologies (.NET, Javascript/Java, C/C++, TAL).

Thank you, any info is appreciated!!

Am using both right now.

Nexus will support more repo types with the free version than Artifactory

but

Artifactory will have more repos support in the paid tier

Nexus now support Conda natively (since a few weeks ago if memory serves)

Artifactory will be quite pricey, especially if you plan on having tests and staging instances to test out your config and/or custom plugins as you will have to shell for at least a Pro instance (3K+USD).  Nexus will allow you to test most of these with the free version.

JFrog has exemplary support but a bit pushy with their sales.  They will have help around though to get you started should you need it.

I have not tried Nexus support or sales force so I cant comment (the free version was plenty enough for our needs).

I found the plugin framework to be more usable with Artifactory though it felt a bit limited.  Nexus seems more powerfull but it can be difficult to upgrade/remove plugins from a working instance.  On that front I much preferred Artifactory.

Overall we did go with Artifactory, personally I find them to be pretty much equivalent.  It will depend a bit on the actual technology you need supported (pypi, nuget, maven etc).


Different download speeds from AWS S3 CLI vs Console/Websites I'm experiencing much slower traffic for the exact same files when downloading (or uploading) them via the AWS website vs. downloading them via the CLI.

Is there a known reason for this?

Is there something that can be done about it?

TL;DR
Try reducing the number of concurrent connections used by awscli to 1 using this command: aws configure set default.s3.max_concurrent_requests 1

You could be experiencing an issue with the number of concurrent connections that the awscli is opening, if you are using the aws s3 commands (not aws s3api).

In the cli user guide under the Managing Objects section you will see that all high-level upload commands on an S3 Bucket beyond a large object threshold (awscli default 8MB) will use a multipart upload.

I couldn't find anything about the use of concurrent connections when using the web console, but looking at the output of lsof I could see my browser was only opening a single connection for a ~500MB file download, whereas using aws cp 10 concurrent connections were being opened. So this would appear to be the key difference in behaviour - at least with my browser and test case.

Running a quick, dirty test with the default (10) connections and then 1 concurrent connection, my download completed in the following times:

s3.max_concurrent_requests=10   8m22.965s
s3.max_concurrent_requests=1    6m47.772s

So it seems the overhead of the extra connections can sometimes cause slower downloads/uploads over my network, rather than making them faster, which could be what you are experiencing.

You could also try tweaking some of the other defaults around how multipart operations are handled which may yield better performance over your network, listed here: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html.


pod to pod can not be reached through service clusterIP running in the same minion few days ago, i am facing a strange issue which is "a pod can not reach another pod running in the same minion/host through servide ClusterIP whereas they can reach each other skipping going through service clusterIP"

i am running kubernetes 1.9.6 on top of openstack (ubuntu 16.04 hosts) with:

cni 0.7.0
cri-containerd 1.0.0-beta
flannel 0.9.1
etcd 3.0.13

the other scenarios were working fine. even between pods running in different hosts, they can reach each other through service clusterIP.

what is going on here !!! any support will be more than welcome

The response from the called service is not DNAT'ed back to set src to the service clusterIP. So the response packet is going directly via cni0 bridge and not getting into iptables, preserving pod's IP.

to fix this i added the following option to kube-proxy and issue was gone.

--masquerade-all=true


Unable to connect to remote service using AWS VPC Peering I have a question regarding some AWS infra I'm trying to build.  Due to legacy constraints, I have a service running in a remote VPC that I need to access, and I have a new service I'm developing in another VPC.  the reason the 2 VPCs are separate is because the newer VPC is managed by Terraform and I am planning to migrate all the services into Terraform but it's going to take a long time, so for the present time I want to connect the 2 VPCs using AWS VPC Peering (I own both VPCs and both services, I can control anything in this infrastructure, except for the remote service code and deployment details which is a black box).

The remote service I believe is deployed as a single EC2 instance, the new service is deployed using ECS Fargate with an ALB on top.  There are 3 security groups involved; the remote instance security group, the ECS security group, and the ALB security group.

I looked at the documentation here as well as a bunch of related pages, and did the appropriate steps.  In particular, I did:

Create the VPC Peering connection
Accept the peering connection
Add entries to both VPC routing tables where the destination is the other VPC's CIDR block and the destination is the ID of the peering connection
Add a TCP exception to the remote service security group to allow incoming traffic from the ECS security group (by security group ID), and vice-versa
Check my Network ACL to make sure I have an Allow rule to accept traffic

For some reason, when I try to make my request over the peering connection, the request gets thrown into the void and never returns.  It's not logged on the remote server meaning that I don't think it ever arrives.  I spun up an EC2 instance inside the ECS security group and tried to call the remote resource manually (using curl) and that didn't work either.

I've followed all the guides I could find and can't figure this out, but I'm a newbie to AWS so maybe there's something I missed.  Does anyone have any other ideas?  Thanks.

How are you trying to connect to the remote service? - IP address, default internal/external DNS name or custom DNS name private/public hosted zone?
Please compare the output of nslookup to the remote service in both the VPCs. If the IPs resolving are different, the traffic is trying to go through the public internet and not the peering connection.
It would most probably be the need to either share the private hosted zone with the peered VPC or share internal hostnames across the peering connection


Should we name names in a blameless post-mortem/retrospective? We want our post-mortems to be blameless.

Recently someone included this statement in a post-mortem report:

On DATE, PERSONMAME did code a fix (link), but forgot to merge it to master

It was suggest that to be "blameless" we shouldn't include PERSONNAME.  The suggested replacement was:

On DATE, the code fix (link) was created but the engineer failed to merge it to master.

In a blameless culture, nobody should be worried about having their name listed in a report.  In the first example, it feels blame-full to point out their name.

On the other hand, in the second example, someone could follow the link and see who the engineer was. This seems to be hiding someone's name without really hiding it.  It feels like something we'd do in an organization that is having a problem with blame/shame.

What should we do?

Usually for a blameless postmortem, the best idea is to go further than the human error (which for a proper 5 whys should not arise, that's rule 11, but we're just humans :)) and complete the 5 Whys with the 5 Hows.

To follow on this particular case for a blameless postmortem I'd go few step further whith those iterations (For the sake of the example) of How possible follow up interpretation (possible answers):

How this fix has been left unmerged after creation ? (it wasn't reviewed and didn't follow the usual path)
How did it miss review ? (That was urgent and straightforward, review wasn't necessary and decision was taken to deploy)
How this process has been defined ? (It wasn't)
How was the bug report closed without merge (No validation/no defined process for this case)
How to prevent the problem arise again ? (Define the process and set validation/review even for quick patches)

Now you don't have someone who have forget to merge something, you have an analysis of how it was possible and more important, you have something actionnable to improve which is the main, if not the sole, reason to do a postmortem.

A more detailled criticism of the 5 whys (which as you show end often with a Who instead of a Why) can be found on O'reilly's blog post The infinite hows

To address the specific point of naming in the analysis, I'd argue neither the name nor the fix itself are of value, wording the step in the analysis as "An engineer wrote a fix and didn't merge it" without any link would be enough for the report.


Error starting cloned VM in Vsphere Client I cloned a VM in Vsphere Client as described here. When I try to start it, it unexpectedly reboot over and over itself. I get this error in console:

Call "EnvironmentBrowser.QueryTargetCapabilities" for object "45-envmgr" on ESXi "192.168.100.172" failed.

Which also shows another status message saying:

The object has been deleted or not created completely.

Have a look at this page which documents EnvironmentBrowser for clues of which setting to edit.

Possibly something has a relative path that is now modified, or a resource that's fully used up or no longer available (or no longer valid, but that makes little sense if it's the same VM host).


This job is stuck because you don't have any active runners online I try to use Gitlab-CI and create the runner, but when jobs start, the job with the needed runner doesn't start:

But this runner exists in the list:

How to use a runner?
And an additional question: all works fine with the "automatic" runner when I only create .yml file. Why do I need to create some other runners and why runners needed?

Runners are machines (typically containers) where your jobs run. Gitlab offers shared runners, and you can also bring in your own runners (machines that you register against Gitlab to run your jobs).
The main difference between Gitlab shared runners and the ones you bring is that the shared runners are, well, shared, so sometimes you have to wait for a bit for a job to start. You can also bring more powerful machines so that your jobs run faster.
You can find some docs on runners here.
In your case it seems that the runners have a problem, looking at the exclamation sign next to the name, which should be a green ball. Maybe hovering over the sign will give you some clues into what's happening.


Can't change Windows Docker storage location (The parameter is incorrect) I'm trying to move the Docker storage location to another drive. This is my configuration file:
{
  "registry-mirrors": [],
  "insecure-registries": [],
  "debug": false,
  "experimental": false,
  "data-root": "E:\\test"
}

I also tried to use "graph" instead of data-root (probably that was they to do it in the older versions) without any luck. I also gave read/write permissions to Everyone just in case. After restarting the daemon, I see the Docker directories and files are created in that folder, however when I try to pull the image, I'm getting following error:

Executing task: docker build --pull --rm -f "Docker\BuildTools\Dockerfile" -t msbuildssdt "Docker\BuildTools" <
Sending build context to Docker daemon  5.632kB Error response from
daemon: chtimes \?\E:\test\tmp\docker-builder614293845\coverage: The
parameter is incorrect. The terminal process
"C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -Command
docker build --pull --rm -f "Docker\BuildTools\Dockerfile" -t
msbuildssdt "Docker\BuildTools"" terminated with exit code: 1.

It's working well if I change the data-root value to any folder located on the C: drive.

The issue was that the E: drive was formatted with exFAT partition. After I formatted it to NTFS it started to work.


How to make PM2 as a autostart service? I want to make my PM2 as a startup service, because for example when my server loses power or something happens to it, it does not automatically restart the PM2 service.

When configuring PM2 is finished and it's working properly, you must run two commands to make your PM2 as a autostart service and enjoy that.
1) Run 'pm2 save': It's save all of your PM2 configure to a file.

admin@server:~$ sudo pm2 save

2) Run 'pm2 startup': This command creates a system service and runs your last change saved with the first step.

admin@server:~$ sudo pm2 startup

So, Run the first command when you changed the configuration to store the last changes.


SSH connection to jenkins server I have a absh script which connects and exexutes programs from my local to remote server and now this should be executed from jenkins build so for this jenkins is taking continous changes from git for this script but my build is failing with below error:
Warning: Identity file /c/Users/D35N/Downloads/GitHubProjeckt/Automation/GitHubProjeckt/newharbortestkey not accessible: No such file or directory.
Host key verification failed.
Build step 'Execute shell' marked build as failure
I have tried to set ssh connectivity in my jenkins server for this remote server incredentials and ssh plugins but using my key i am getting an error cant connect to the server.
when i run the script from local it works fine and my ssh connection gets established properly using SSH key.
jenkins server is also integrated or installed on my kuberenets cluster which is my remote server in this case and bash script should execute on this kuberenets cluster.
Can anyone please suggest how this problem can be solved?

Your configuration refers to a file that is local on your system, but does not exist on the jenkins server.  The error message is trying to tell you this.  /c/Users/D35N/Downloads/GitHubProjeckt/Automation/GitHubProjeckt/newharbortestkey is not on the jenkins server.  You need to make this file available on the jenkins server by copying it up there, and it probably shouldn't be in your personal Downloads folder.
You may want to configure your script to look for a series of files and use the first one that exists.  This will let you have the file at different locations on your development box versus the server and it will still work in either.


GitLab: Make commit in (shared) CI runner and push to repository/open merge request I want to run a scheduled pipeline on the shared runners that is able to open a merge request on my repository. I want to run a script that checks if updates for my pinned packages exists and if yes it should update these and open a merge request with a new branch. I am using https://gitlab.com and not a self hosted GitLab instance.
However pushing to my repository failed because whether using ssh or https the authentication is denied.
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.com/<username>/example.git/': The requested URL returned error: 403

or
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://<username>:@gitlab.com/<username>/example.git/'

My gitlab-ci.yml looks something like this
doSth:
      image: image

      script:
        - chmod 700 doSth.sh
        - chmod 700 autoUpgrade.sh
        - ./doSth.sh
        - ./autoUpgrade.sh

My autoUpgrade.sh looks something like this.
#!/bin/bash

if [[ `git status --porcelain` ]]; then
    git config --global user.email "email@example.com"
    git config --global user.name "example"
    export BRANCH_NAME=change-sth-`date +"%s"`
    git checkout -b ${BRANCH_NAME}
    git add -u
    git commit -m 'change sth.'
    # I tried using ssh and https
    # chmod 0400 $CI_DEPLOY_KEY
    # GIT_SSH_COMMAND="ssh -i ${CI_USER_TOKEN}" git push --set-upstream origin ${BRANCH_NAME}
    git push https://${CI_USER}:${CI_USER_TOKEN}@gitlab.com/${CI_PROJECT_PATH}.git ${BRANCH_NAME}
else

Is it possible to push to a new branch in my repository using the shared runners? And if yes do you know what I am doing wrong?
echo "Nothing changed"
fi

CI token used for code checkout is not allowed to do writes to the git repository (unlike GitHub Actions). You need to generate a new personal token and add it to secrets.
See our integration project here as a sample: https://gitlab.com/taleodor/sample-helm-cd
Note, if you search through StackOverflow, you may find several other ways to achieve this but in any case you need to explicitly pass valid credentials that are able to write to the repo.


Where may I need Docker Compose as Kubernetes won the container orchestration war? I found Docker Compose just release 1.28 on 2021-1-20 but as Kubernetes won the container orchestration war I am not sure where and why I may still need Docker Compose, under with case Docker Compose is a better option?
I have a web app that consists of a php docker, a nodejs docker, a nginx docker. We host it at aliyun cloud service. In 2018 aliyun used docker swarm so we used docker compose but in 2020 we had switched k8s so I was wondering if it is still worth to learn docker compose ?
--- update ----
I found these articles were helpful

Why do I need Compose if I already have Kubernetes? a docker blog published in 2018
Docker Swarm vs. Kubernetes for Single-Host Implementations
https://stackoverflow.com/questions/47536536/whats-the-difference-between-docker-compose-and-kubernetes an answer is what I do currently:

If you are networking containers within the same host go for docker
compose.
If you are networking containers across multiple hosts go for
kubernetes.

And that is what docker compose document said  https://docs.docker.com/compose/#common-use-cases

Docker compose is still a comparatively easy and light-weight tool to run multiple docker containers with service discovery.
Kubernetes is a metaphorical elephant.  Great if you need to smash Roman lines and win the battle, but overkill for a minor skirmish, and definitely more trouble than it's worth if you're trying to fight a small battle in the mountains.
If you have a fairly large app with multiple microservices, then Kubernetes is worth every penny.
But when you're trying to run just a couple of containers, especially if it's only on a single host (i.e. your frontend + backend and maybe a local instance of RabbitMQ), docker-compose is a much simpler/faster way to orchestrate it.  Kubernetes is like using a cannon to kill a mosquito when a flyswatter will do.
Example use cases where docker-compose works just fine:

Local development environments running on your machine.  Even if you run Kubernetes in production, there's likely a lot of orchestration around getting it to work which would make it too complex for a single machine.
Simple apps without a lot of components
Single tenant apps which run on a single host
Deploying docker-based apps on bare VMs or physical hosts


How to safe restart Jenkins? I need to make some configuration changes on our Jenkins instance that will involve restarting Jenkins a couple of times. However, our developers are committing frequently enough that I haven't seen Jenkins without jobs running in three days.

Is there a native way (either through the GUI or via command line) to safe-restart Jenkins? IE: wait for current jobs to finish before going down, and keep track of queued jobs to start once Jenkins comes back up.

I know there's a plugin but in order to install it I need to restart Jenkins...

Visiting https://youjenkinsdomain/safeRestart will set it in the mode when it waits for jobs to stop and then restarts.


monitor for docker stats similar to htop? I'm in the need for this and my google leaded me to this post on SO

I guess it's helpful to boost the beta for our sister site DevOps Stackexchagne here.

I quote the original question

docker stats gives a good overview of the cpu/mem/net io/disk io as text.

Does there exist a htop like monitor that shows the docker stats?

bcicen on githup with tool ctop is working for me
ref. https://github.com/bcicen/ctop


Unable to get jupyter/scipy-notebook running on docker I pulled the docker image and is up to date. When  I  run it it says that it's going to be accessible at 8888 on  my localhost but when  I try to  access it resource can't be found.

Here is the output:

sudo docker run jupyter/scipy-notebook #jupyter
[sudo] password for tech:
Execute the command
[I 06:16:54.964 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[W 06:16:55.137 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 06:16:55.195 NotebookApp] JupyterLab alpha preview extension loaded from /opt/conda/lib/python3.5/site-packages/jupyterlab
[I 06:16:55.211 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 06:16:55.211 NotebookApp] 0 active kernels
[I 06:16:55.211 NotebookApp] The Jupyter Notebook is running at: http://[all ip addresses on your system]:8888/?token=8bc0bf76526f10c078aad705f48f80598fe17aade9897db8
[I 06:16:55.211 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 06:16:55.212 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=8bc0bf76526f10c078aad705f48f80598fe17aade9897db8

what could I  be  doing wrong?

It turned out to be an issue with port mapping. After mapping local machine port 8888 to docker's 8888 I could launch the program in browser.

sudo docker run -p 8888:8888 jupyter/scipy-notebook


straight python alternative to ansible Can anyone recommend an alternative to Ansible that lets one write straight Python code? In my group, we're finding that expressing pretty simple logic in Ansible ends up being too time-consuming and ugly to specify. Also, the bunny trails that you end up going down to debug simple Ansible problems take way too much time.

Sometimes it takes hours to do (or debug) something that would take 30 seconds in straight Python.

The nice things about Ansible are:

A nice set of concepts like roles, hosts, tasks, and playbooks.
A nice library of preexisting modules.

.. but if one were looking for something that preserves some of those nice things from Ansible, but allows one to write straight Python for all of the configuration and logic, are there any recommendations?

pyinfra was designed for this purpose. Fabric can also be used to script configuration.


Cassandra persistence with Docker - is an example of a comprehensive, up-to-date and facts-based guideline for a deployment architecture available? Being new to this topic, I have found quite many pros and cons about making data persistent while working with dockerized Cassandra. However, most of it was posted in 2016.

Now, I understand that it can depend on the specific case, but already googling the very question reveals a lot of debate which appears sometimes opiniated (and maybe also historically related to earlier versions of Docker and Cassandra which might have been improving over time and will in future, but maybe there is a conceptual flaw).

So I post this question to differentiate the pros and cons in a non-opiniated way in order to support finding a source or an answer which allows an architectural decision based on facts (or draws a working example, even for a specific case).

Myself is quite biased to Docker but as a DevOps engineer I think I have to learn which things work and which not, also understanding the WHY behind both.

Pros or seems to be quite ok but to prove:

Production, yes or no: is there an real-world example of a large-scale Cassandra in a Swarm/Kubernetes environment? I thought, initially, yes?
Guidelines like "Cassandra and Docker - Lessons learnt" (Slide 20)
Docker Official Image packaging for Cassandra on GitHub

Cons/Criticism:

Articles like "Thou shall not run a database in container"  or "Why database are not for containers" (related yc debate)
A colleague told me, Cassandra would store data associated with IP addresses and this makes usage of Docker/Cassandra not possible if you need data persistence - can't yet google up this but if there is no workaround this will make persistence impossible in my opinion.

Side note. Sometimes they say also, "don't do it in production #1, but integration environment is okay also without persistency #2". I want to learn more about #1 through this question and can't agree with #2 because sometimes you can have a huge effort to prepare already the sample data you need in the integration environment because the production data is yet bigger/complexer.

Since you posted your original question, Datastax has come out with dockerized images of datastax enterprise, along with a comprehensive guide. I'd start with this reference and maybe check out this older "Best Practices" doc, although some details might have changed.


Bitbucket Pull Request Builder Configuration Jenkins I need to integrate BitBucket with Jenkins, so that whenever a PR is opened in BitBucket, it will trigger a build in Jenkins. Can someone help describe the exact configuration step that I should follow in BitBucket and Jenkins to achieve this?
This is my configuration so far:

You can install 'Bitbucket Server Webhook to Jenkins' in Bitbucket.
https://marketplace.atlassian.com/plugins/com.nerdwin15.stash-stash-webhook-jenkins/server/overview

Once install you will need to enable the 'Post Receive' Hook on the required repository.

Remember to enable 'poll SCM' for the Jenkins Job.

Update
You can test the configuration of the plugin here:


Overcommiting CPU Resources in Kubernetes Let’s assume the following scenario:

I have a cluster with 100 CPU cores and 100 Pods each having a request of 1 CPU.

If I schedule the 101st Pod if a request of 1 CPU this will no longer be schedulable, as per my understanding.

Now, further assume that I as cluster operator have observed that pods on average only consume .3 CPU in reality. It would be beneficial if scheduling further pods instances was possible.

Can I somehow apply a global correction factor that disregards CPU requests to some degree and allows me to overcommit the available resources?

You should be able to use the flag spec.containers[].resources.limits.cpu to set your desired state on a deployment.

The official docs here and here have some examples of how to specify this in the yaml.  You can set a global default along with min/max limits:

spec:
  limits:
  - default:
      cpu: 1000m
    defaultRequest:
      cpu: 300m
  - min:
      cpu: 300m
  - max:
      cpu: 1000m


Docker's value and scenarios in native Windows environment AFAIK, typically Windows enterprise stack components are quite monolitic services, like Windows IIS and MS SQL Server. Still, there are "Windows containers".

Are they indeed runnable in containers and this makes sense?
Are there other Windows native containerization scenarios?

In this example Microsoft refers to a ".NET-nanoserver" image. The base image is there a ".NET runtime" - I consider this is something else than an OS base image.

Are they indeed runnable in containers and this makes sense?

According to this documentation it is possible to run windows in docker. Yes it could make sense as this would be comparable to the positive effects of running linux in docker. Note that:

  Disclaimer: This lab is still in work, and is based off of the blog,
  but you can test and leverage the example walkthroughs now, if you
  want to start experimenting. Please check back as the lab evolves.

Whether it makes sense: From a Linux perspective it does not as a Windows container will be significantly larger than a Linux one at the moment. From a Windows perspective it does not make sense as well as Windows itself is not as lightweight at the moment as Linux. Imagine that one runs 50 war files on a dedicated Windows server at the moment. If one would use Windows Docker containers for that purpose then a dedicated server should be used as a base image, each war file should be run in a separate container. This means that the number of required resource would grow exponentially. I assume that the creators of the native Windows Docker container take this into account and that this issue will be solved in the future.

  Are there other Windows native containerization scenarios?

What is the definition of a microservice? One could also argue that running SQLServer and IIS are microservices if one compares it with running IIS and SQLServer and other Windows services on a Windows server instead of separately. If one compares it with running Linux in docker then one could use a windows server as a base image by defining FROM windows in a dockerfile and run tomcat in a Windows container instead of running it on a dedicated server.


terraform chef provisioning - cannot validate certificate for 10.90.239.223 because it doesn't contain any IP SANs While configuring the chef provisioner I am running into the following error after running 'terraform apply'

chef_environment.terraform_01: Post https://10.90.239.223/organizations/mykitchen/environments: x509: cannot validate certificate for 10.90.239.223 because it doesn't contain any IP SANs

I tried this

openssl req -x509 -newkey rsa:4096 -keyout chef.key -out chef.pem -nodes -days 365 -config req.conf

sudo chown opscode:opscode chef.*
sudo mv chef.* /var/opt/opscode/nginx/ca/.

sudo chef-server-ctl stop
sudo chef-server-ctl reconfigure
sudo chef-server-ctl start

my req.conf

[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C=US
ST=mystate
L=mycity
O=DevOps
OU=myname
emailAddress=myemail@mycompany.com
CN = 10.90.239.223

[ req_ext ]
subjectAltName = @alt_names

[alt_names]
IP = 10.90.239.223
DNS.1 = 10.0.90.45
DNS.2 = 10.90.50.8
email = DevOps-3@10.90.239.223
URI = https://10.90.239.223/

but I am still getting

chef_environment.terraform_01: Post https://10.90.239.223/organizations/mykitchen/environments: x509: cannot validate certificate for 10.90.239.223 because it doesn't contain any                                       IP SANs

In my chef provisioner config I also have

ssl_verify_mode = ":verify_none"

So far I am stumped.  What am I missing?

Answered on Slack, the likely first issue here is that the output from openssl req is not a cert, it's a CSR. But there are some other lingering issues here too. It is recommended to use something like Certstrap to do internal cert generation rather than raw openssl commands as the modern standards are quite fiddly.


From 6 months to 16 seconds? grokking DevOps speedup data from DevOps report by DORA and Puppet Labs Puppet Labs' annual DevOps reports are a very nice and representative source of information, AFAIK. Indeed, with Kim Gene on board they are THE industrial gurus I'd say.

Now I have a question to the following data in this year's report (page 21).

Can we multiply the data for 2016 and 2017?

For example, take the lead time for changes.

2555x440=1124000 # more than a milion times faster?

Does this mean then, if in 2015 it took 6 months (say 180 days real time i.e. 4320 hours) to deliver a bugfix, then

in 2016 it took just under 2 hours,
and in 2017, 16 seconds?

Is the warp speed delivery speed real now? In 2014, management was very sceptical about even 30x speedup. Now I'm sceptical.

Question: this must be absolute minimum values for very lightweight microservices. That is, as the bugfix commit arrives..

you really count from this moment
you have no compilation step
the Docker image is very small
you have really fast (or, fastest you can get) infrastructure.

Right?

I wouldn't exactly calculate it like that. It doesn't necessarily mean that the actual verification time for a changeset is decreasing that much.

The verification process itself can be (and often is these days) a processing pipeline in which the average/effective verification time per changeset can, indeed, decrease to insanely small amounts.

For example the processing pipeline can process multiple changesets simultaneouly. If you have, let's say, 60 proposed changes, each one of them touching a different file in a repository, you can combine them into an equivalent single uber-changeset touching 60 files. You could then process this uber-changeset through a verification step that takes, let's say, 1h. If the verification passes you obtain a speedup factor of x60 and an effective verification time per changeset of 1 min, even if the actual verification time was 1h. Granted, you'll lose some of the speedup when the verification fails and you need to perform bisections to identify the changeset at fault, but that's a different story.

Or you can have each of the changesets verified independently, through 60 1h verifications performed in parallel. Same effective verification time, but with a whole lot higher verification resource usage.

The point, I guess, is that there are ways of building processing pipelines which can greatly reduce the effective per-changeset verification times and still keep very large scale projects agile.

Another point to consider is the branching strategy. Those 6 months weren't the actual verification time. Most of that time was simply spent navigating through the branch spagetti and fighting the integration hell, just to propagate the fix into the proper shipping branch - effects of the waterfall development model which was considered the norm at the time.

Today CI/CD (the proper one, not the CI Theatre) and Trunk Based Development eliminate all that wasted time/effort and side-stepping, with the fix being virtually ready for delivery right from the moment it is integrated in the trunk - no unpredictable massive branch merging obstacles ever stand ahead of shipping.


Add item(s) to yaml array only if defined (ansible) How do I write a yaml file such that it defines an array with a list of values, and it skips those values from being added to the yaml array if the given value is undefined?
Specifically, I'm trying to define an array of IP addresses to which I want a service (sshd) to bind to, including the ipv4 and ipv6 addresses. If there is both an ipv4 & ipv6 address, then the array should contain both values. If there is only an ipv4 address or only an ipv6 address, then it should contain one of those values.
This doesn't work, as it will produce an error "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'address'"}
    ssh_listen_to:
      - "{{ ansible_default_ipv4.address }}"
      - "{{ ansible_default_ipv6.address }}"

This also doesn't work, as it will end-up adding a line ListenAddress  with no argument, that breaks the sshd server (this may be a shortsighted design of my ansible role, but that's out-of-scope in this question).
    ssh_listen_to:
      - "{{ ansible_default_ipv4.address | default('') }}"
      - "{{ ansible_default_ipv6.address | default('') }}"

How can I update the above yaml block such that it will skip adding the values to the array if the value isn't defined? Note that this is being done in my playbook's vars: section, so no tasks-based solutions please.

Create the list in Jinja2 and convert it, e.g.
- hosts: localhost
  gather_facts: true
  tasks:
    - set_fact:
         ssh_listen_to: "{{ _ssh_listen_to|from_yaml }}"
      vars:
        _ssh_listen_to: |
          {% if ansible_default_ipv4.address is defined %}
          - {{ ansible_default_ipv4.address }}
          {% endif %}
          {% if ansible_default_ipv6.address is defined %}
          - {{ ansible_default_ipv6.address }}
          {% endif %}
    - debug:
        var: ssh_listen_to|type_debug
    - debug:
        var: ssh_listen_to

gives
TASK [debug] ********************************************************************
ok: [localhost] =>
  ssh_listen_to|type_debug: list

TASK [debug] ********************************************************************
ok: [localhost] =>
  ssh_listen_to:
  - 10.1.0.27

If you put the name of the variables into the list, e.g.
    vars_addr:
      - default_ipv4
      - default_ipv6

the task below gives the same result
    - set_fact:
         ssh_listen_to: "{{ vars_addr|
                            map('extract', ansible_facts, 'address')|
                            select('defined')|
                            list  }}"


Pull request and merge actions in CI/CD The CI/CD pipeline supports pipelines per branch but no pipeline for feature branches is triggered automatically. There are pipelines deployed for the following branches: development, master. The respective pipeline is triggered onCommit event (codeCommitRepo.onCommit('OnCommit', {})  so whenever there is any code push to the branch. We are planning to use git-flow as the branching strategy but don't exactly see the advantages of having another branch called staging, in addition to development, master.

Is there a use of having a third branch like "staging" when using CI/CD?

The CI/CD pipeline looks as follows:

At what stage should the PR from the feature branch be created?
At what stage should the PR be merged into the development branch?
At what stage should the PR from the development branch to master be created?
At what stage should the PR be merged into the master branch?
What if the development pipeline fails? Should the code be manually unmerged to get development back in the stable condition?
What if the master pipeline fails?
Is the code being merged in the branch after the pipeline succeeds?

I would like to understand what the flow should look like starting from a feature branch.  (P.S: Not interested in Trunk based development strategy)
Also, the pipeline is coming from https://github.com/awslabs/aws-simple-cicd and is written using AWS-CDK

In my opinion there should be a staging branch , which holds the production candidate code.And instead of a Test env , have a staging env which will be complete replica of Prod env. In the Staging env the QA can perform their testing and other checks.

Dev env =  only for developer.
Stage env =  Replica of Prod and holds production candidate build/code , where QA can perform their testing and simulate Prod.

Having a Staging branch has its benefit as you can have code which will merge to master and deploy to prod , and the developer can experiment/work on develop branch without worrying of any accidental merge to master directly.

Once someone completes a feature or fixes a bug/error or implementation is done and testing needs to be performed.

same as above.

Once the code has been passes the test's and QA verified on staging env and No new   feature are to be added to Prod , it can be merged to master.

No new feature are to be added to Prod , QA passed and test cases passed, it can be merged to master

That's where the Staging branch will help , as even if code fails on stage as it is supposed to be complete replica of prod env. we can revert or commit the fix again without worrying of our master branch.


kubectl asking for username (k8s cluster created via kops) I created a kubernetes cluster using kops and deployed it in AWS (not EKS manual cluster using kops).
Then I export in my local machine the configuration like
kops export kubecfg --state ${KOPS_STATE_STORE} --name ${NAME}
Then running any kubectl command asks for a username, even kubectl version
 kubectl version
Please enter Username:

PS: If I ssh into bastion host then ssh into k8s master node, kubectl works fine and its configuration is referring to local user created.
My local config created from export command into ~/.kube/config looks like:
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: fakedata=
    server: https://api-test-intern-doesntmatter.eu-central-1.elb.amazonaws.com
  name: test-internet.k8s.local
contexts:
- context:
    cluster: test-internet.k8s.local
    user: test-internet.k8s.local
  name: test-internet.k8s.local
current-context: test-internet.k8s.local
kind: Config
preferences: {}
users:
- name: test-internet.k8s.local
  user: {}

And with the config above I cannot do any kubectl command.
Where can I find the username and password for kubectl?
I have set authentication as RBAC for the cluster and no limitations.
I read all article here https://kubernetes.io/docs/reference/access-authn-authz/authentication/ and still do not understand what is the default username password why am I asked for such credentials I haven't created any role or user.

I solved this temporarily with not the safest solution, but is working
When creating secret with kops named admin, can use that to generate user field in exported kops config by adding also
--admin in the export command


Managing Data bags in different environments Hi I got stuck with one of the scenario, I have two environments Dev and Prod. I need to pass different values in dev and different vales in prod. So how can I manage it using one Data bag. this is my general Data bag in dev environment

I am using this data bag in Dev environment, so I have to pass different username, passwd and jdbc password for prod environment. How can I do this? I am planning to do in below manner

{
 "id": "some_data_bag_item",
 "production" : {
 "id":"Databag_item",
 "username": "username",
 "passwd": "****",
 "jdbc_password": "****"
}
 },
  "testing" : {
    "id":"Databag_item",
    "username": "username",
    "passwd": "****",
    "jdbc_password": "****
  }
 }

Is this is a right way? Do we need to add anything else in recipe? Do chef will find out which environment it is automatically?

https://www.digitalocean.com/community/tutorials/how-to-use-roles-and-environments-in-chef-to-control-server-configurations

  For instance, if a node is in the "production" environment, you could
  want to run a special recipe in your "nginx" cookbook to bring that
  server up to production policy requirements. You could also have a
  recipe in the nginx cookbook meant to configure special changes for
  testing servers.

Example

name "web_server"
description "A role to configure our front-line web servers"
run_list "recipe[apt]", "recipe[nginx]"
env_run_lists "production" => ["recipe[nginx::config_prod]"], "testing" => ["recipe[nginx::config_test]"]


Cancel sibling snapshot dependency on failure I have a simple build chain configure in TeamCity:

                         build ->
create-base-image ->                      -> docker publish
                         test  ->

I have the "On failed dependency: cancel build" option enabled. So if either build or test fails, the queued docker publish step will be cancelled.

However if build fails, the test step continues to run all the way through and visa versa.

I would like to cancel all sibling tasks if any of them fail. Or if a dependent step is cancelled (docker publish) I would like all currently running dependencies of that to be stopped.

Someone asks the same question here:
https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206926235-Cancel-other-snapshot-dependencies-if-one-dependency-fails-

But I'm not sure if the linked YouTrack issue is actually solving this problem.

I was unable to find a way to configure the build to behave like I wanted. Instead I needed to start using the TeamCity API to achieve this.

In the build.sh and test.sh file that TC runs, I extended the failure function (simple bash function we call whenever our build script fails):

teamcityFailure () {
    . ./cancel_build_chain.sh
    echo "##teamcity[buildProblem description='$1']"
    exit 1
}

This now calls off to this new cancel_build_chain.sh script, which looks something like this:

#!/bin/bash

teamcityProgress () {
    echo "##teamcity[progressMessage '$1']"
}

teamcityProgress "Build step has failed, attempting to cancel the rest of the build chain..."

BUILD_ID=$TEAMCITY_BUILD_ID # passed in from build chain configuration

# Use the current failed step ID to query for the snapshotDependency "from:" here - which returns the final "docker publish" step
FINAL_STEP_DATA=$(curl --silent --request GET \
    "https://teamcity/app/rest/builds/project:MyProjectName,snapshotDependency:(from:(id:$BUILD_ID),includeInitial:true),defaultFilter:false" \
    --header "Content-Type: application/xml"")

# From the "docker publish" step, parse the XML and fetch out all dependent step IDs. One will be this current step, the others will be its sibling steps
# We do this twice, once for queued steps, and once for currently running steps. This is because we need to cancel running vs queued builds differently.
QUEUED_DEPENDENT_STEP_IDS=$(echo $FINAL_STEP_DATA \
 | grep -Eo '<snapshot-dependencies.+<build id="[0-9]+".+state="queued"' \
 | grep -Eo 'id="([0-9]+)"' \
 | grep -Eo '[0-9]+')

# Now remove the current step ID
QUEUED_DEPENDENT_STEP_IDS=${QUEUED_DEPENDENT_STEP_IDS/$BUILD_ID/}

# trim leading / trailing whitespace
QUEUED_DEPENDENT_STEP_IDS=QUEUED_DEPENDENT_STEP_IDS | sed 's/ *$//'

RUNNING_DEPENDENT_STEP_IDS=$(echo $FINAL_STEP_DATA \
 | grep -Eo '<snapshot-dependencies.+<build id="[0-9]+".+state="running"' \
 | grep -Eo 'id="([0-9]+)"' \
 | grep -Eo '[0-9]+')

# Now remove the current step ID
RUNNING_DEPENDENT_STEP_IDS=${RUNNING_DEPENDENT_STEP_IDS/$BUILD_ID/}

# trim leading / trailing whitespace
RUNNING_DEPENDENT_STEP_IDS=RUNNING_DEPENDENT_STEP_IDS | sed 's/ *$//'

for DEP_ID in $QUEUED_DEPENDENT_STEP_IDS
do
    teamcityProgress "Cancelling queued build step $DEP_ID"
    curl --silent --request POST \
        "https://teamcity/app/rest/buildQueue/project:MyProjectName,id:$DEP_ID" \
        --data "<buildCancelRequest comment='Another part of the build chain failed so this sibling step was cancelled.' readdIntoQueue='false' />" \
        --header "Content-Type: application/xml"
done

for DEP_ID in $RUNNING_DEPENDENT_STEP_IDS
do
    teamcityProgress "Cancelling running build step $DEP_ID"
    curl --silent --request POST \
        "https://teamcity/app/rest/builds/project:MyProjectName,id:$DEP_ID" \
        --data "<buildCancelRequest comment='Another part of the build chain failed so this sibling step was cancelled.' readdIntoQueue='false' />" \
        --header "Content-Type: application/xml"
done

Not super elegant but it gets the job done and prevents build agents sitting there running build steps for builds which have already failed.


DynamoDB vs RDS for a counter store? I have multiple lambda functions, which a file passes through for processing, and I want to store the status of each computation in a counter, for better logging and monitoring.

Current plan is to log it like this:

file-name  Lambda1  Lambda2 Lambda3
Name1      Done     Done    In-process

And, I'll be updating the values as Received, In-process and Done under the respective Lambda functions for the file.

So, two doubts:

Is this a good practice? If not, then is there a better alternative for the same?
If it is, then what would make a better/cheaper store? RDS or DynamoDB?

Why not something like Redis? (can use it through ElastiCache). Hard to answer this without knowing the operational requirements of this project. Is this data strictly for logging/research, or is it operational (i.e. this "table" is consulted in order to know which lambda to invoke).

Dynamo vs. RDS - depends how spiky your traffic is. Dynamo is very scalable and will give you a guaranteed throughput. But if you traffic can grow 10x over hours (spiky) Dynamo might be too expensive, since you'll need the maximum throughput provisioned or else you'll get throttled.

RDS is basically managed RDBMS on an instance type that you choose, so it can handle larger spikes, but ultimately not as scalable as Dynamo.

Again, I would consider Redis, for it's simplicity (seems like you just need a KV/counter store), and incredible speed/efficiency. If it's just for logging, it's probably good enough.


What is a good logging practice for distributed tasks? I have the following setting:

  Create multiple workers, do a computation  and terminate them after the
  computation is done.

So, every-time it’ll be a different instance running the task, so each host will have its own a log file, this will result in a huge list of files.

Is it a good practice? If not, what would be a better way for logging the task processing in this particular use-case?

PS: My infrastructure is serverless. So, for now, I am logging to (AWS)CloudWatch. But, please answer the question independently of AWS, and suiting a serverless setup as much as possible.

"Serverless" mostly just means you've got relatively simple microservices, generally just a little webapp or a single function that is automatically connected to a REST frontend. The same concepts apply as you would use for a more traditional web services: usually some mix of remote syslog and ElasticSearch writers.

Networked or remote syslog has been around for a long time and has a fairly robust set of tools around it. You would have to run the central syslog server(s) but the protocol is very simple and there are pure client libraries in every language that you can use for sending logs. One common problem with remote syslog is that it has traditionally been based around UDP. This means that under heavy load, some log messages may be lost. This could be a good thing, helping avoid a cascade overload, but it is something to be aware of. Some newer syslog daemons also support a TCP-based protocol, but client support is less unified so just do your research.

More recent but very popular is logging to ElasticSearch. This is mostly useful because of the Kibana dashboard and Logstash tooklit (often called ELK, ElasticSearch+Logstash+Kibana). Amazon even offers a hosted ElasticSearch option, making it somewhat easier to get started. ES uses a relatively simple REST API, so any language with an HTTP client (read: all of them) should be okay with logging to ES but make sure you are careful with blocking network operations in cases of partial system outages (i.e. make sure your app won't get stuck in a logging call that will never succeed and stop servicing user requests).

More complex logging topologies are bounded only by your imagination, though these days you'll see a lot of use of the Kafka database/queue/whatever-you-want-to-call-it as a nexus point in very complex log distribution systems.

On the "serverless" side, you'll generally want to integrate with these systems directly at the network level, so sending log data directly to syslog or ES from your service/function, rather than writing to local files (though maybe echo to those too for local debugging and development).


Execute the whole Playbook serially Adding serial: 1 to a Play will execute that one Play one host at a time. Is there something similar for the whole playbook?

That is, all plays will execute only in one host, and when they are finished then all plays will execute on the next host, an so on.

I think Ansible serial works only for the entire play, have you tried to use roles ?

For example:

- hosts:
    - web
    - db
  serial: 1
  roles:
    - { role: nginx, when: ansible_os_family == 'Debian' }
    - role2
    - role3


Can one use lando.dev for a production environment? OpenSource community developed free lando.dev looks great for a local development platform on one's own machine. I'm certainly looking to evaluate it for my own setup.

But, given there are benefits in having a development environment that matches production, if one is using lando.dev in development, then could one use it in production.

Sure, one might want to go on to use Kubernetes in production, but perhaps this is more than what one needs in the early stages of a project and therefore lando.dev is more suitable and more accessible.

From "the horses mouth":

https://twitter.com/DustinLeblanc/status/1220351066178510848

  We don't recommend Lando in production. The container images we use
  aren't optimized for that and we don't test an open-to-the-internet
  scenario. It's exclusively meant for local dev


Making thousands of microservice/dockers with different IPs I need to simulate many connections. Currently using curl.
What is the best way to deploy thousands of such requests with different IPs using some script?

Linux, without docker

In Linux, you can create arbitrarily many IP address for each interface. A comment in https://serverfault.com/questions/328146/max-number-virtual-ip-addresses-per-nic reports success with 2000, and hearsay of 5000 successful IPs.

So. Pick any Linux box on your intranet, which could of course be a VM, and create as many IPs on its single ethernet interface (MAC address) as you wish.

Say you're using the 10.0.0.0/8 private network, then you can execute lots of these:

ifconfig eth0:1 10.0.0.2 netmask 255.0.0.0
ifconfig eth0:2 10.0.0.3 netmask 255.0.0.0
...
ifconfig eth0:260 10.0.1.6 netmask 255.0.0.0
...

You should be able to use curl --interface eth0:3 (or maybe --interface 10.0.0.3) to use a specific source address. For wget, it's --bind-address=10.0.0.3 etc. Start the curls in the background ("&") and they will run more or less concurrently - or look for some test driver (different question) to run them for you.

This will give you lots of requests from separate IP addresses, but all conveniently running on one machine.

Docker, trivially

With docker: just make an image, as small as humanly possible, which connects to your server. You can use curl without any special options.

Then fire up your 1000 docker containers, just like you usually would. Each will automatically get an IP address on the internal docker network (172.x.0.0/16 by default).

https://stackoverflow.com/questions/21799382/is-there-a-maximum-number-of-containers-running-on-a-docker-host indicates that it is possible with hardware that is not too uncommon today; a naive 1000 docker containers used roughly 3 GB of RAM for them. That question also has some hints on options you can use to decrease the memory needs. Aside from memory, there won't be too much overhead, i.e. CPU wise it should comparable.

Docker, with special networking

docker network opens up manually created networking for you. You should be able to cobble something together (just check the individual --help pages, it's all pretty self explanatory), and combine that with the ifconfig approach explained above. This would mean that you only need one container for your curls, and start 1000's of them inside that container.

I would try to just run your 1000 containers, first, though, unless you are limited in the RAM department, for ease of use.

About the curl...

Whatever you do, don't just start a single curl in each instance, but have each runner be an endless loop of back-to-back curl calls. This way, you avoid long gaps inbetween where your individual jobs are being started and stopped; this is obviously especially important for the "1000 docker containers" variation. In the answer linked above, they note that it took them some good amount of minutes to start up those thousands of containers, so you do not want to do that over and over again.

And you might as well get rid of curl altogether and create small networking app in the language of your choice (Perl, Ruby, C, Java...) to avoid process startup overhead.


Docker daemon error on joining swarm node - could not find local IP address - unknown port Consider the following setup:

Three hosts in the same network:

Host A - SLES
Host B - Ubuntu
Host C - Ubuntu

Then, the following has worked:

Init swarm on A
Join swarm at B, make B a manager

The following is however not working:

Join swarm at C as worker with token issued from A
Join swarm at C as worker with token issued from B

In both cases we get the same error:

Error response from daemon: could not find local IP address:
dial udp: address udp/2377: unknown port

Honestly, I have even no idea how to interpret this message.

Any ideas?..

UPD. nmap results for ports 2377,4789,7946 (hint from Felipe in a comment below) - no idea what are they good for but now it is clear that something is wrong with host C.

Host A (Primary manager)

PORT     STATE  SERVICE
2377/tcp open   unknown
4789/tcp closed unknown
7946/tcp open   unknown

Host B (Secondary manager)

PORT     STATE  SERVICE
2377/tcp open   unknown
4789/tcp closed unknown
7946/tcp open   unknown

Host C (should become a worker)

PORT     STATE  SERVICE
2377/tcp closed unknown
4789/tcp closed unknown
7946/tcp closed unknown

Host D (successfully added as a worker)

PORT     STATE  SERVICE
2377/tcp closed unknown
4789/tcp closed unknown
7946/tcp open   unknown

Finally, with host B there is the actual problem.

swarm init => OK; port 2377 is open
swarm leave => port is closed again
swarm join => error "port not found"

Could you please try the commands above between all hosts ?

INSIDE HOST A:

nc -zv HOST_B_IP_ADDRESS 2377,4789,7946
nc -zv HOST_C_IP_ADDRESS 2377,4789,7946

INSIDE HOST B:

nc -zv HOST_A_IP_ADDRESS 2377,4789,7946
nc -zv HOST_C_IP_ADDRESS 2377,4789,7946

INSIDE HOST C:

nc -zv HOST_A_IP_ADDRESS 2377,4789,7946
nc -zv HOST_B_IP_ADDRESS 2377,4789,7946

As explained in https://docs.docker.com/engine/swarm/swarm-tutorial/#open-protocols-and-ports-between-the-hosts, some ports must be available between hosts in order for Docker Swarm to work correctly.

The error reported sounds like a port related problem.
Maybe is there another application in one of these hosts already using one of the ports needed by Docker ? Maybe a firewall configuration ?


Why should I store artifacts? Why should I store artifacts? I'm doing it but i don't know if it still make sense.

My builds are fairly quick
I do git tag versioning
None of my builds depends on another. (all independent micro-services)

Any compelling reason to keep this "Artifactory code" in my pipelines?

One argument in support of storing artifacts is that it promotes consistency when deploying across different environments.
From Continuous Delivery by Humble and Farley:

Every time you compile the code, you run the risk of introducing some difference. The version of the compiler installed in the later stages may be different from the version that you used for your commit tests. You may pick up a different version of some third-party library that you didn’t intend. Even the configuration of the compiler may change the behavior of the application.

By only building artifacts once, we make sure that the production code is exactly the same as that which has gone through testing. In very small codebases with simple pipelines, it might not make too much of an impact, but it's still best practice to only build once.


How to prevent that tags will be overwritten in docker private registries? This is a follow-up to this question about preventing tag overwrite in docker hub.

When one pushes a tag to a private docker registry and the current tag exists then it will be overwritten. How to prevent this?

I have considered using a digest and other solutions, but this does not guarantee overwrites as one is still able to tag an image using a certain digest. I really would like to see an error and an exit code of 2.

It is possible to prevent the overwrite of tags by listing the existing tags using https://github.com/jessfraz/reg and exit the pipeline if there is a duplicate.


One time database update process as part of applying a patch I have a PostgreSQL database that requires a server side script to be executed (client prefers php) to:

Perform some intelligent data modifications to records that will be
identified  only during the update.
Update specific functions for a
dynamically identified list of schemas.
Generate a static rollback
script/data based upon what is being modified.

I am hoping to integrate it with gitflow like process to allow for some control and visibility as in the past these types of changes were done manually.  I've been looking for something similar to libraries like phpunit to avoid reinventing.

Are there any scripting libraries or tools that can serve as a starting point.

This is usually called a migration, try yii2, Laravel or doctrine

Mostly these frameworks will manage the schema in yml, XML or PHP and automatically calculate schema changes and apply them.

For data changes, you would package up a number of database queries or even program code into a migration, which usually has a forward/up step and a backward/down step.

Migration status is usually stored as a single value in a separate database table


git reflog is showing plain text password used We are using Jenkins Freestyle Project to push the changes on the remote server. We are executing shell script on remote host using ssh for it. To pull the changes on remote host, we are using origin url with git username and git password. The credentials should not be visible in plain text in the url that's why we have stored them in variables using 'secret text(s) or file(s)' option of 'Build Environment'.
At the Jenkin's end it is working as it is expected. The git credentials are not visible to the users who are using Jenkins for other projects but we are facing issue on the remote server where project was deployed. The remote server is showing git credentials in plain text. Any user with ssh access of the remote server is able to run the git reflog command in the project directory.
Port 22 cannot be opened on the server where gitlab is deployed so we cannot use ssh keys method to create the build in Jenkins. We can use only http method to pull the changes.
Is there any way so we could implement to avoid showing the git credentials in plain text in the project directory.

You could use the git credentials.helper feature
https://git-scm.com/docs/gitcredentials


See which Roles contain a GCLOUD Permisson I'd like to see wich Roles in iam yield a certain permission.
In other words, for a given permission (like billing.accounts.updateUsageExportSpec) I want to know a list of Roles, that satisfy this permission.

Looks like only the roles/billing.admin:
https://cloud.google.com/iam/docs/understanding-roles#billing-roles


Terraform: Is there a concise syntax for specifying multiple tags for a resource? To attach multiple tags to a resource, I currently use multiple tag blocks:

resource "aws_autoscaling_group" "instance" {
  ...

  tag {
    key                 = "Name"
    value               = "${var.cluster_prefix}"
    propagate_at_launch = true
  }

  tag {
    key                 = "Owner"
    value               = "${var.tag_Owner}"
    propagate_at_launch = true
  }

  tag {
    key                 = "Project"
    value               = "${var.tag_Project}"
    propagate_at_launch = true
  }
}

It works, but I wonder if there is a more concise syntax. It is also a bit error prone as you have to specify propagate_at_launch for each tag.

Update: The question was written before Terrafrom v0.12 came out. Since then the support has improved with Dynamic Nested Blocks (see my answer below).

For an auto-scaling-group this is the most concise syntax available.

For most other resources you use the tags syntax which looks like:

tags {
  Key1 = "value1"
  Key2 = "value2"
}


How to run database migrations in an auto-scaled group We are just about to move from a monolithic architecture to an auto-scaled group and I'm not sure how I should be running my database migrations (Laravel).

I am thinking that a script will run when a new box comes online, this will git pull my latest code down. Should this script also execute my database migration? I am not sure how I can get it run on only one box?

How are you doing your deploys?  Nuke + rebuild the ASG (whether one node at a time, or by replacing the whole group at once), or do you have a script that redeploys all of your active nodes?

And also, how are you triggering your deploys?

Ideally you want to be running/triggering deploys from some form of CI server like Bamboo or Jenkins.  If you're doing this, you can run your DB migrations from the build server (or its component build agent instances).

This would allow you greater control over your environment, as you could, for example, restore DB from snapshot if the deployment fails.


Meaning of metrics in Google Cloud Console I have a 3-node kubernetes cluster with 3 n1-standard-1 :1 vCPU, 3,75 Go RAM, in a gcloud projet. Recently, I ran a test that created quite a few jobs (about 150). Half the jobs were running an hello world script, the other half were failing due to configuration error, and those jobs kept retrying by restarting new pods.

In my console I observed the following graph:

It seems that my cluster is capped at 3% of maximum utilization, if I read the units correctly. What bothers me is that in my test, the amount of pods/utilisation should not be constant, so I was not expecting the utilization to be a flat line like this.

Also, I noticed about about 30 pods were in unschedulable state, so my initial thought was that I reached maximum utilization of the instance.

EDIT

It is clear now that my test is too poorly designed, so I cannot conclude about maximum utilization so far.

It seems that my cluster is capped at 3% of maximum utilization, if I
  read the units correctly. This makes no sense at all, why isn't it
  capable of reaching 100% ?

Why do you think it does make no sense at all?

  I ran a few tests that created quite a few pods

What is a few tests?

  (each pod running basically an hello world script)

What does this script do? Print helloworld? If that is the case then it could be possible that the cluster will not use 100%.

By the way, how large is the cluster? How many agents? How many memory? So if the cluster possesses 1024GB of memory then it could be the case it is using 3%.

In summary, to indicate whether something makes no sense at all a number of facts is required:

content of the helloworld script
characteristics of the cluster, e.g. totol cpu, memory, type of disks, storage capacity, number of agents, masters
definition of few tests and few pods. How many tests were run, frequency, e.g. 10/s, how many pods

If you would really like to loadtest a k8s cluster, kubernetes has some documentation about this, e.g. this document explains how to load test a k8s cluster and in this case force autoscaling.


How to pick a right hardware configuration setup(server) for MERN stack I'm learning programming(MERN stack) and along with that basic DevOps(linux, containers etc). I've read a lot about software configuration setups(with NGINX, and without, etc), but can't find anywhere regards how well the MERN stack will behave on different server hardware configuration setups.
I will probably host my app on Linode, for various reasons(price, anti-ddos etc), but I guess it will applicable for alike services too(digital ocean etc).
So, is there a way to approximately know before deploying, how much my app will need resources? Are there any assumptions I can make regards MongoDB's usage(of RAM, for example?), for example?
Any info will be valuable for me, thank you.

Since this is for your learning, it shouldn't tax your hardware incredibly. If you are hosting I would with a free student account if you can. But if you have a linux laptop all of this should work well on it. You're not writing apps that are getting 10,000's of connections and transforming complex data. If you had an enterprise app I would indeed split off the DB from the frontend etc.


Setting up CI/CD for an AWS CDK app using AWS CodeBuild/Deploy/Pipeline I'm trying to setup a CI/CD pipeline for a dotnet app which uses AWS Lambda and AWS CDK for infrastructure. The source is on github and we have a Jenkins pipeline which runs the tests and publishes the artifacts. I want to use the artifact and deploy (or better use Code Deploy)

Can I use CodePipeline to run cdk deploy?
How can I use CodeDeploy to do a dotnet test and dotnet publish? and then pass on the artifact to CodePipeline

It may be helpful to look at the different AWS developer services as you are trying to achieve your goal in the wrong services:

Running the specific dotnet testing and publish steps are not done in CodeDeploy.  Rather, they would be done in AWS CodeBuild, in the AWS Pipeline, or directly on your Jenkins Server.
AWS CodeDeploy is for taking your build artifacts and deploying them "to a variety of compute services such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers".
AWS CodePipeline is for modeling an automated workflow.  This service integrates with Jenkins and GitHub.

With these services in mind I think what you are looking for is to start with a new CodePipeline.  This will allow you to start building a model that specifies where you are running your dotnet tests (Jenkins or on AWS) and what to do with your artifacts (publish through Jenkins or AWS CodeDeploy).  This can be accomplished with AWS CDK.


What are the best practices of merging dev to master? We have 2 branches (master/dev) + some feature branches and 2 environments (production/dev). Our team is a very small people. What are the best practices of merging dev to master?

Every morning?
After integration tests?
How about merging master with dev and then testing (hotfixes) then merging master to dev?
Some release branches?

I don't want to create anything too complicated.

You can use "git flow" or more simple "gitlab flow".

Our small team merge into master on release or sprint ends.


One Jenkinsfile or multiple? So I have a Jenkinsfile defining a build pipeline and then a Jenkins job (not pipeline) with a very simple deployment script for our Docker stacks.

Seeing that Jenkinsfiles can become as complex and powerful as one's coding skills are (seeing that they can groovy fairly well), would I have everything in one Jenkinsfile or have multiple Jenkinsfiles?

Consider this theoretical process:

Development
> Build artefact > Build Docker image
> Deploy Docker container to testing > tests & QA
> deploy to staging > have client check it
> deploy to production

Would all of that go into one Jenkinsfile and run different steps of the pipeline according to user input like

"which stages should be performed?
Please specify:
Build artefact,
build Docker Image, ..."

This would result in a complex Jenkinsfile but you'd only have one single file.

Or should this process be split into multiple Jenkinsfiles with multiple Jenkins jobs/pipelines for each Jenkinsfile to do a part of the process?

It really comes down to personal preference.

One additional tool you might not be aware of are shared libraries for Pipeline.  These allow you to quickly write custom Pipeline steps or factor out common Pipeline code without writing a Jenkins plugin in Java.

Between multiple Pipeline jobs and shared libraries, there are many ways to split up your job's code and there's no one right answer on how to do it.  One suggestion I have is to determine what steps in your process are "atomic" - that is, what are the smallest group of steps that you envision admins/ops/devs/etc. might need to run on their own?  Each "atomic unit" should then become its own job.

For instance, let's say you're automating a deploy process, and your process in a general sense looks like this:

build
deploy to dev
deploy to prod

Then I would create three jobs, one for each bullet point.  This gives you some nice advantages:

You can re-run each job individually in case something fails instead of having to restart the entire process over from the very beginning.  Deploy to prod failed?  You only need to re-run the deploy to prod job.  You don't need to rebuild or re-deploy to dev.
Each stage in the process is separated out so you can see how often it succeeds or fail.  Some stages in your process may be more robust or fragile than others; this kind of separation gives you insight into that.
This level of abstraction makes it easy for non-ops people to perform ops tasks (if that is something you want/need at your org).  When everything is monolithic, the only thing non-ops people can do is run the entire process from the beginning.  When everything is split up into tiny jobs, you need intimate ops knowledge to know which pieces to run in what order.  By separating your jobs into independent, easy-to-understand, appropriately bite-size chunks, a non-ops person only needs to press a single button to kick off an automated ops process.


How to detect what interface has private IP in Ansible I plan to reconfigure iptables on our servers (mostly CentOS 7, some CentOS 6) with different network configurations, some on 1Gbit interface, some on 10Gbit, some with VLANs and some with configured LACP.

I want to use Ansible, but couldn't find how to detect what inteface has private address. Private address we could get from inventory as ansible_ssh_host.

Is there any Ansible module that could help? Or I should write shell/Python script?

I checked all solutions on StackOverflow and ServerFault but haven't found anything.

Peter's module not working in CentOS and I write my own module:

#!/usr/bin/python
# encoding: utf-8

import subprocess
from ansible.module_utils.basic import *
from netaddr import IPNetwork

def get_priv_interface_info_for_ip(ip):
    network_config = subprocess.Popen(["/usr/sbin/ip", "-o", "addr", "show"], stdout=subprocess.PIPE)
    result = network_config.stdout.read()
    interface = ""
    network = ''
    ip_addr = ip
    for str in result.split("\n"):
        if 'inet ' in str:
            if ip in str:
                interface = str[3:str.find(" ", 3)]
                network = IPNetwork(str[str.find("net", 3) + 4:str.find("/", 3) + 3]).cidr

    return interface, ip_addr, network

def get_pub_interface_info_for_ip(ip):
    network_config = subprocess.Popen(["/usr/sbin/ip", "-o", "addr", "show"], stdout=subprocess.PIPE)
    result = network_config.stdout.read()
    interface = ''
    network = ''
    ip_addr = ''
    for str in result.split("\n"):
        if 'inet ' in str:
            if ip not in str and '127.0.0.1' not in str:
                print str
                interface = str[3:str.find(" ", 3)]
                network = IPNetwork(str[str.find("net", 3) + 4:str.find("/", 3) + 3]).cidr
                ip_addr = str[str.find("net", 3) + 4:str.find("/", 3)]

    return interface, ip_addr, network

def main(argv=None):
    if argv is None:
        argv = sys.argv

    fields = {"ip": {"required": True, "type": "str"}}
    module = AnsibleModule(argument_spec=fields)
    my_ip = module.params['ip']
    ifname_priv, ip_priv, network_priv = get_priv_interface_info_for_ip(my_ip)
    ifname_pub, ip_pub, network_pub = get_pub_interface_info_for_ip(my_ip)

    module.exit_json(changed=True, ifname_priv=ifname_priv, ip_priv=my_ip, network_priv=str(network_priv), ifname_pub=ifname_pub, ip_pub=ip_pub, network_pub=str(network_pub))

if __name__ == '__main__':
    main()

It works for servers with two network interfaces and provide information about public and private network interfaces.

Actual version of this module could be found in my GitHub.


Is it possible to have pair of IPs in Ansible? I have so many HA machines that one of them is primary and the other is standby.  I have this playbook that copies a config file from primary and then from standby on local machine then use those files as input for a python script:

- hosts: primary
  remote_user: root
  tasks:
  - name: copy the config of primary to local
    fetch:
      src: /data1/cronjobs/gen_ddl
      dest: /AnsibleDir/primary
      flat: yes
- hosts: standby
  remote_user: root
  tasks:
  - name: copy the config of standby to local
    fetch:
      src: /data1/cronjobs/gen_ddl
      dest: /AnsibleDir/standby
      flat: yes
- hosts: 127.0.0.1
  connection: local
  remote_user: root
  tasks:
  - name: run python script
    shell: /AnsibleDir/driver.py
    register: myshell_output
  - name: copy the output to a local file
    copy:
      content: "{{ myshell_output.stdout }}"
      dest: "/AnsibleDir/output.{{ host }}"

and I use this command:
ansible-playbook -i /Ansible/hostfile /Ansible/python.yml --extra-vars "host=test1"

in hostfile I have this configuration:

[primary]
192.168.107.69
[standby]
192.168.107.70

I think that I should edit this file every time I want to run my playbook. so is it possible to have so many pairs of IPs?
for example something like this:

primary:1.1.1.1,standby:1.1.1.2
primary:1.1.1.3,standby:1.1.1.4
primary:1.1.1.5,standby:1.1.1.6

or can I assign each pair the variable I want instead of writing it in my command every time? like:

primary:1.1.1.1,standby:1.1.1.2,host:test1
primary:1.1.1.3,standby:1.1.1.4,host:test2

I searched a lot but I could't find anything and I'm new to ansible. Any suggestions?

Q: Is it possible to have a pair of IPs in Ansible?

A: Yes. It is possible. For example, it might be useful to create groups in the inventory with all pieces of information.

[primary]
1.1.1.1
1.1.1.3
1.1.1.5

[standby]
1.1.1.2
1.1.1.4
1.1.1.6

[ha]
1.1.1.1 stdby=1.1.1.2
1.1.1.3 stdby=1.1.1.4
1.1.1.5 stdby=1.1.1.6

It's possible to simplify the fetching of the configuration.

- hosts: primary, standby
  remote_user: root
  tasks:
  - name: copy the config to local
    fetch:
      src: /data1/cronjobs/gen_ddl
      dest: /AnsibleDir/conf

Without flat: yes the configuration files will be stored in

/AnsibleDir/conf/{{ inventory_hostname }}/data1/cronjobs/gen_ddl

Then it's possible to reference the configuration files by the inventory hostname. For example groups['primary'] (secondary the same way). Take a look at myshell_output and select the data for content and dest from the items

- hosts: 127.0.0.1
  connection: local
  remote_user: root
  tasks:
  - name: loop primary config
    command: "/AnsibleDir/driver.py
              /AnsibleDir/conf/{{ item }}/data1/cronjobs/gen_ddl"
    register: myshell_output
    loop: "{{ groups['primary'] }}"
  - name: copy the output to a local file
    copy:
      content: "{{ <SELECT-FROM-ITEM> }}"
      dest: "/AnsibleDir/output.{{ <SELECT-FROM-ITEM> }}"
    loop: "{{ myshell_output.results }}"

To reference primary/standby pairs use group ha. For example the play below changes the configuration both at primary and standby for all hosts in ha.

- hosts: ha
  tasks:
    - name: Change configuration at primary
      lineinfile:
        path: /AnsibleDir/conf/{{ inventory_hostname }}/data1/cronjobs/gen_dd
        regexp: <ADD-REGEXP>
        line: <ADD-LINE>
    - name: Change configuration at standby
      lineinfile:
        path: /AnsibleDir/conf/{{ hostvars[inventory_hostname].stdby }}/data1/cronjobs/gen_dd
        regexp: <ADD-REGEXP>
        line: <ADD-LINE>

  Q: I need to find the differences of configuration files in 2 machines. they need to be the same. In python script, it will find the differences and fix them. So I need to have both configurations on my local then run the python script for them.

A: Fix the configuration files in 2 machines

- hosts: ha
  tasks:
    - name: Fix primary and standby config
      command: "/AnsibleDir/driver.py
                /AnsibleDir/conf/{{ inventory_hostname }}/data1/cronjobs/gen_ddl
                /AnsibleDir/conf/{{ hostvars[inventory_hostname].stdby }}/data1/cronjobs/gen_ddl"


Kubernetes namespaces with RBAC: sufficient for tenant isolation in a public environment? Initially namespaces have been created to enable logical partitioning of a single Kubernetes cluster in order to prevent naming conflicts.
Over time, this changed. While this article from 2016 sais

Any user or resource in a Kubernetes cluster may access any other resource in the cluster regardless of namespace. So, if you need to protect or isolate resources, the ultimate namespace is a separate Kubernetes cluster [...]

With RBAC it's now possible to set permissions on a specific namespace and thus restrict authenticated users to specific namespaces and their resources.
So, if I get this right, the quote from above is outdated. With RBAC in place it's possible to make sure a user or resource may only access stuff in their own namespace. Is that correct?
I struggle to find recent documentation that explicitly sais that it's fine use namespaces (with RBAC) to seperate tenants from accessing each others stuff. Does anyone use namespaces and RBAC to seperate (unfriendly) tenants? Is there any clear statement about this from the CNCF?
If so, purely from a security perspective, what level of isolation are we talking about here? Clearly, the cluster IP would be accessible to all tenants and even with quotas in place I can imagine simple DOS attacks that might break a single cluster. Simple RBAC misconficutions might leak data between tenants and of course there's also the risk of implementation error and future exploitation. But if we consider those things "acceptable", are namespaces really a good tool for tenant isolation?

There's an active SIG devoted to multitenancy so I have a hard time believing that nothing has been written about it in 4 years.
Namespaces are only part of the answer. That isolates kubernetes objects impacting other kubernetes objects, but you also need to worry about shared resources like the ingress namespace. If containers share worker nodes, then there are resource limits to enforce. And the default network model used by kubernetes is flat with DNS exposing services and containers in other namespaces and an open network policy allowing any container to talk to any other container. The kubernetes API is also shared in a multi-tenant environment, allowing a DoS to the kubernetes infrastructure itself. And if users are allowed to run containers with added capabilities, privileged access, host bind mounts, host namespaces, and probably many other features, they can escape the container isolation and get root access on the host running the containers. None of this is trivial to implement securely.


How can I issue a Let's Encrypt certificate with Ansible when 'challenge_data' cannot be found? I am attempting to use the Ansible acme_certificate module to acquire a Let's Encrypt TLS certificate.

I have successfully generated the private keys, CSRs, etc, and then started the challenge using the following task:

- name: Start challenge
  acme_certificate:
    acme_version: 1
    account_key_src: /etc/letsencrypt/account/account.key
    challenge: http-01
    csr: /etc/letsencrypt/csrs/{{ item }}.csr
    dest: /etc/letsencrypt/certs/{{ item }}.crt
    fullchain_dest: /etc/letsencrypt/certs/fullchain_{{ item }}.crt
  with_items:
  - www.example.com
  register: acme_challenge

I then attempt to satisfy the http-01 challenge using:

- name: Implement challenge
  copy:
    content: "{{ acme_challenge['results']['challenge_data']['www.example.com']['http-01']['resource-value'] }}"
    dest: "/var/www/html/{{ acme_challenge['results']['challenge_data']['www.example.com']['http-01']['resource'] }}"
    owner: root
    group: root
    mode: u=rw,g=r,o=r

However, this results in the following error:

The error was: 'list object' has no attribute 'challenge_data'

I have tried accessing the acme_challenge variable in different ways, for example using {{ acme_challenge.challenge_data['www.example.com']['http-01']['resource'] }}, as well of various other combinations of square brackets and dots.

I've also copied the example of accessing the variable directly from the Ansible documentation, here and here, but that gives the same error.

After around 3 hours of trying I cannot read the challenge data at all. I've used a debug statement and I can confirm that it's all there properly, I just can't read the variable with this syntax.

Is this a known problem, or does anybody have a suggestion?

This was caused by the fact that I was starting the Let's Encrypt challenges using a with_items loop. From here:

  When using register with a loop, the data structure placed in the variable during the loop will contain a results attribute, that is a list of all responses from the module. For a more in-depth example of how this works, see the Loops section on using register with a loop.

The solution is to have a separate task for each Let's Encrypt challenge, rather than having them all in one.


How to save groovy output to pipeline variable I have the below code in the jenkisn pipeline:

   stage ("amd_distribution_input_transformation"){
        steps{
            script{
                    amd_distribution_input_transformation url: params.DOMAIN_DESCRIPTOR_URL, secret: params.CERDENITAL_ID
                }
            }
        }

amd_distribution_input_transformation.groovy content:

def call(Map parameters)
{
    def CREDENITAL_ID = parameters.secret
    def DOMAIN_DESCRIPTOR_URL = parameters.url
    sh '''
        python amd_distribution_input_transformation.py
      '''
    }
}

in the amd_distribution_input_transformation.py some code is running, and at the end, it returns object named 'artifacts_list'

my question is, how can I assign the return obect from groovy file to pipeline variable.
BTW, if it helps, I can write the output to json file from the python code (and here I'm stuck on how eventually assign that file to the pipeline variable)

Have you tried setting returnStdout to true in the sh() command per the Jenkins documentation?

You can then assign that output to a variable within your pipeline:

def artifactsList = sh(label: 'Running amd_distribution_input transformation', returnStdout: true, script: 'python amd_distribution_input_transformation.py')

and check what kind of object that variable is, and act on it accordinly

echo(artifactsList.getClass())