Recommendations about release convention or standard for Python scripts? I have a lot of python scripts which I release to a python virtual environment on a remote machine. The virtual environment is rarely updated. The scripts are pulled from a git repository and a makefile is used to deploy them into the virtual environment directory.

My question: is there a more normal way to deploy the scripts?

I would like a process that at least remembers which files were deployed for cleanup upon the next release. I've read a bit on pip packaging, but I'm not sure if it fits our setup. The scripts need to be in a common location where they can be called by other applications. The OS is SUSE Linux.

One could consider to package the scripts. Depending on the distribution, e.g. ubuntu, windows, centos one could create a ppa, nuget or rpm respectively. Once a newer version is installed, the package manager will remove the previous version automatically.

As yum or zypper could be used to install packages on OpenSuse one could consider to create an rpm by using either spec files or using fpm.


How do I work around the fact that AWS SQS is not HIPAA compliant? I have a use case where data from S3 is queued into AWS SQS, which is in turn connected to CloudWatch, whose metrics will be triggering AWS Lambda.

However, I want the architecture to be HIPAA compliant. So, I have come up with this idea:

Once my S3 bucket gets a file,
Fire up a Lambda function, which does hashing/name scrambling of the files, and copies to another S3 bucket (via aws cp)
Connect the bucket with the hashed/scrambled names to the SQS queue

Is this a good and secure practice? Or is there a better workaround? (Would be more than happy if I can send encrypted keys of S3 to SQS. But not sure if I can or if it is possible)

According to Amazon AWS

  Customers may use any AWS service in an account designated as a HIPAA
  account, but they should only process, store and transmit PHI in the
  HIPAA-eligible services defined in the BAA. There are ten
  HIPAA-eligible services today, including AWS Snowball, Amazon
  DynamoDB, Amazon EBS, Amazon EC2, Amazon Elastic MapReduce (EMR),
  Amazon Elastic Load Balancing (ELB), Amazon Glacier, Amazon Relational
  Database Service (RDS) [MySQL, Oracle, and PostgreSQL engines only],
  Amazon Aurora [MySQL-compatible edition only], Amazon Redshift, and
  Amazon S3.

source: https://aws.amazon.com/compliance/hipaa-compliance/

This means that as long as you are not storing or transmitting PHI in SQS, just the information about where this PHI is being stored - you probably can pass an audit reg. HIPAA compliance.

In the architecture you describe, the SQS queue does not need to include any PHI content. This would make it comply with the above statement.

More information about HIPAA compliance on AWS is available in this whitepaper from January 2017 - https://d0.awsstatic.com/whitepapers/compliance/AWS_HIPAA_Compliance_Whitepaper.pdf

Specifically SQS is mentioned and explained in the HIPAA FAQ - https://aws.amazon.com/blogs/security/frequently-asked-questions-about-hipaa-compliance-in-the-aws-cloud-part-two/.

update: As of May 1st 2017, SQS is now HIPAA compliant. https://aws.amazon.com/about-aws/whats-new/2017/05/amazon-simple-queue-service-sqs-is-now-a-hipaa-eligible-service/


Is there a way through which I can control type of load balancer like round robin , least-connections etc using helm charts? I am researching different types of loadbalancing methods available through kubernetes .I want to implement & see before deciding on one .I'm stuck at how to actually define typeof-loadbalancer.If it isn't possible through helm  can anyone suggest other options please .

Well there are 2 possibble options.

One is to create a LoadBalancer type for each service you want to be exposed. In case you have just one service that's great. That's rarely the case of course :)

If you build your k8s cluster properly ( having Prometheus-Operator doing the monitoring for example ) most likely besides the business logic service/s, you would want to expose  internal cluster valuable services like Grafana.

That's why even if you have just 1 service, for future extension of such type of need the best would be to use an Ingress resource. It providees a way to expose multiple services through a single IP address —  HTTP level (network layer 7).

You could play with HTTP headers essentially and do some path/host routing.

You express the definition for your object and helm would do the installation. Another option is to use a tool like ArgoCD, basically sticking to GitOps policy.

A fundamnetal knowledge is to be aware that the Load Balancer/Ingress would forward the traffic to the service, not the pods. Services are abstracted entities by kube-proxy, by default kube-proxy uses a round-robin algorithm.

https://kubernetes.io/docs/concepts/services-networking/service/

Regarding the definition, it follows the standart K8s scheme [ apiVersion, kind, metadata, spec fields]. Example would be an Ingress created from AWS-Ingress-Controller.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: management
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
  - host: grafana.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 80
        path: /*


Problem on gitlab-runner: cannot lock ref [...] Another git process seems to be running in this repository I got a problem on a gitlab-runner after pushing several commits:

Running with gitlab-runner 11.4.0 (8af42251)
   on GITLAB-RUNNER 58c31dfa
 Using Docker executor with image registry.git.rwth-aachen.de/luechow-group/inpsights/gnu:7.3.0 ...
 Pulling docker image registry.git.rwth-aachen.de/luechow-group/inpsights/gnu:7.3.0 ...
 Using docker image sha256:8c48f31ac37d239cf7c072961068e85dd0af222604461b7c0df905a45e900877 for registry.git.rwth-aachen.de/luechow-group/inpsights/gnu:7.3.0 ...
Running on runner-58c31dfa-project-12102-concurrent-0 via GITLAB-RUNNER...
00:05
Fetching changes...
 HEAD is now at 15dd3f8 .
 error: cannot lock ref 'refs/remotes/origin/develop': Unable to create '/builds/luechow-group/inPsights/.git/refs/remotes/origin/develop.lock': File exists.
 Another git process seems to be running in this repository, e.g.
 an editor opened by 'git commit'. Please make sure all processes
 are terminated then try again. If it still fails, a git process
 may have crashed in this repository earlier:
 remove the file manually to continue.
 From https://git.rwth-aachen.de/luechow-group/inPsights
  ! 62d2cd4..f2bb2fe  develop    -> origin/develop  (unable to update local ref)
    15dd3f8..0c2b081  feature/ElectronSelectorRebase -> origin/feature/ElectronSelectorRebase
 Auto packing the repository in background for optimum performance.
 See "git help gc" for manual housekeeping.
 error: cannot lock ref 'refs/remotes/origin/develop': Unable to create '/builds/luechow-group/inPsights/.git/refs/remotes/origin/develop.lock': File exists.
 Another git process seems to be running in this repository, e.g.
 an editor opened by 'git commit'. Please make sure all processes
 are terminated then try again. If it still fails, a git process
 may have crashed in this repository earlier:
 remove the file manually to continue.
 error: failed to run reflog
 ERROR: Job failed: exit code 1

Which command do I have to use to delete the .lock file in the docker container and to save this change?

The easiest solution for me was to execute

docker container prune

on the runner machine. The command removes all stopped docker containers and the .lock files therein. Running a pipeline from gitlab.com then starts a new container. (Note that all caches will be lost).


How to consolidate multiple servers for CPU and Memory with Kubernetes Setup: I have 10 Photovoltaics based setup where IOT Gateway get data from sensors, devices and send to Kubernetes Cluster. K8S cluster exposed its ActiveMQ broker with NodePort.
Observation: AMQ has been scheduled on Node2 and i observed that CPU usage of this node is around 75%, As we are collecting lot of messages, But other nodes CPU usage is like 8% to 20%. If we put more load this CPU might reach to 100%.
Although we have CPU resource available but we cant use it, How can I do optimal usage of resources?
I read Apache Mesos and DC/OS but i think still its not possible to run a container workload which requests more cpu /memory than available on its single largest physical/virtul host.
Any thoughts in this regard?

Since Pod is running on single node it can use resources from this node only. And CPU/MEM can't be shared here. At least without special software for distributed computation.
In your case you should consider using ActiveMQ Clustering (https://activemq.apache.org/clustering). In that way you will be able to spread load across multiple k8s nodes with ActiveMQ pods running on them.
So use k8s Deployment + pod affinity to be sure pods will be scheduled to different nodes.


How can I get terraforms extern to execute `ssh-keygen -y -f ~/.ssh/id_rsa`? I seem to be confused about how external works. I tried:

data "external" "local_key" {
  program = [
    "ssh-keygen", "-y", "-f ~/.ssh/id_rsa"
  ]
}

This gives me:

  failed to execute "ssh-keygen":  ~/.ssh/id_rsa: No such file or directory

Which presumably happens because ~ expansion doesn't. ssh-keygen -y -f ~/.ssh/id_rsa works normally. So instead I tried invoking bash like:

data "external" "local_key" {
  program = [
    "bash"
  ]

  query {
    "-c" = "ssh-keygen -y -f ~/.ssh/id_rsa"
  }
}

I'm still getting the same issue. I realize I can just pass the output as a var to terraform from the outside, but I'm still curious what the solution is.

So what happens in first case is, as Dan's already said, there's no shell used and as such nothing to expand the ~. Quoting the documentation about program:

  Terraform does not execute the program through a shell

On the second case, bash receive in stdin something like this:

{ "-c": "ssh-keygen -y -f ~/.ssh/id_rsa" }

And this looks like a command block for bash, but -c is not a valid command.

What could work could be this kind of program (assuming no specific input):

jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{"pubkey":$pubkey}'

So something like this should work to get the key in local_key["pubkey"] if I understand the documentation properly:

data "external" "local_key" {
  program = [
    "bash", "-c jq -n --arg pubkey \"$(ssh-keygen -y -f ~/.ssh/id_rsa)\" '{\"pubkey\":$pubkey}'"
  ]
}

There's a need to use bash for a one liner to take advantage of command subsitution. You can also do a .sh script like:

#!/bin/sh
jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{\"pubkey\":$pubkey}'

And call this script in the program parameter.


What are Design Patterns for creating images? In Object Oriented Programming (OOP) there are Design Patterns (DP). DP are best practices for tackling certain problems. For example, a singleton is acceptable to be used in a logger class.

What about applying DPs to creation of images, e.g. docker. In our situation we have the core, i.e. the application, application configuration (customization per customer), and different artifacts per customer.

At the moment, a bash script is used that contains various if-else statements and it is getting more and more a spaghetti.

The general design pattern is one container = one running service - see the initial comment at https://docs.docker.com/config/containers/multi-service_container/

Now, after spending about a year developing a multi-service application relying heavily on docker this is what works for me:

Avoid creating files inside the container unless it is your storage engine - in which case use volumes (recommended by Docker and also common sense) - we use Elasticsearch for that.

If I need multiple custom configurations I am trying to either load it from somewhere (DB, Environment) or I mount the configuration as a volume like

-v config.yml:/usr/src/app/config.yml

In general, we have one service = one repository, but sometimes you have to copy all the dependencies in order to just create one script. So we have one repository where these scripts are aggregated and I change the CMD with an environment variable like this:

CMD cd /usr/src/app && python -u $RUN_SCRIPT

It can be probably improved with a combination of ENTRYPOINT and CMD as described here http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/

Then we use various docker-compose.yml files for getting the setup we need.

For some operations (like DB setup) we either run one container like

docker run --network ournetwork init-profiling

or I can run a command inside that container

docker run -it --network ournetwork init-profiling bash
# run_command.sh
^D

We spent some time minimizing the footprint of containers, they can quickly get bloated. For example, we use a common base image for our python containers. Also, things like pip freeze do make sense only if you use a separate virtual environment for each repository/container.

We do not use swarm/kubernetes for now, so I cannot add any additional guidelines for that.


How should I separate micro service modules for the message part(posts, reply, private) of a social network? I have designed different part of social networking system using micro service and I have reached to the message part of the system. Messages can be private messages (maybe a separate module), the posts on the wall for each user which can be images, text, video (again maybe another module) and reply messages which is again can be a different module.

Now my problem here is that should I design separate modules (service) for each kind of message. Should I consider load of messages as part of this concept of separating modules?

From one view they are all messages that users use to interact with other users in the system.

How should I design micro service for the message part of this social network?

EDIT-1:
Micro-service modules of my project for now:

user
authenticator
notifications (email, push, etc)

Now my social network project has 3 parts for messages:

users' posts
users' private messages
users' replies to other messages

These 3 categories can be 1 module that can manage all these logics. Or 3 different modules user posts, user private messages, user replies.

The question is how should I separate them? One of my concerns is that each category would get more logic and it will lead to a bloated module.

My Final Question: Is it just the role of micro service that separate a module from another or should I take complexity of module into account for the separation?

NB: By messaging I don't mean micro-service module's inter communication, but user's messages in the ecosystem.

From my point of view, you can design microservices just like classes in an application. That is, you are looking for maximum cohesion within a service. In other words, all stuff inside one microservice should work together to achieve exactly one well-defined responsibility, and no other microservice should have any content for the same responsibility.

Furthermore, you want to minimize coupling. That is, mainly, that each microservice should have a very well defined interface, and should be easily replacable in the grand scheme of things. This also means making use of concepts like dependency injection and such.

So as long as you have these two aspects under control - cohesion and coupling - you are fine. The exact separation of concerns (more coarse or more fine-grained) is not that important and will likely differ between different applications.

There are obvious concerns here - for example, it would not make sense to put "Logging" and "GUI" in the same component (ask yourself questions like this: "Would anything else than a GUI use the Logging?" or "Does GUI need to know how Logging is implemented?" or "Does the GUI need to be implemented by the same team that does Logging?" and so on). Another frequent example would be handling of data imports - does that belong into the GUI service just because users usually upload their Excel files through a GUI? Obviously not - you may wish to upload Excel via a REST API or by pulling them from another server in a batch job, etc.

So, in your example, you best sit down and define the responsibilities for every component. Make sure there are as little overlaps as possible. This will usually lead to merging of components you thought to be different; or splitting up of components which turn out to have a well-defined border.


Jenkins Pipeline Kubernetes Agent shared Volumes I have a Jenkins Pipeline using Kubernetes Agent.

I schedule multiple containers in my pod template.

I recently wanted to add volumes that would be shared some of the containers and am having difficulties.

I need to know how to create a volume that will be shared between 2 or more containers in a pod. That volume should mount to a specific path on each container.

Specifics:

Using GCE (Google Cloud Engine) with Kubernetes Jenkins
Using emptyDir volumes

There also seems to be a severe lack of documentation for this. So any direction on that would also be appreciated. The documentation I did find says that the way to do this is to define volumes in the pipeline that will be shared by all containers.

Sample Jenkinsfile:

Below is not my actual pipeline, but it recreates the issue.

pipeline {
    agent {
    kubernetes {
      label 'test-pod'
      defaultContainer 'jnlp'
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: frontend-test
    image: centos:7
    command:
    - cat
    tty: true
  - name: backend-test
    image: centos:7
    command:
    - cat
    tty: true
  volumes:
  - name: sharedvolume
    emptyDir: {}
    mountPath: '/opt/app/shared'
"""
        }
    }
    stages {
        stage('Test Pipeline Configuration') {
            steps {
                container('frontend-test') {
                    sh 'touch /opt/app/shared/test_file'
                }
                container('backend-test') {
                    sh 'ls /opt/app/shared/test_file ; echo $?'
                }
            }
        }
    }
}

Result:

[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘test-pod’ are offline
Agent test-pod-5t0gj-wln8d is provisioned from template Kubernetes Pod Template
Agent specification [Kubernetes Pod Template] (test-pod):

Running on test-pod-5t0gj-wln8d in /home/jenkins/workspace/pipeline-test
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test Pipeline Configuration)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ touch /opt/app/shared/test_file
touch: cannot touch '/opt/app/shared/test_file': No such file or directory
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

To solve this, you need to include the volume mounts in the containers described here:

Sample Jenkinsfile:

pipeline {
    agent {
    kubernetes {
      label 'test-pod'
      defaultContainer 'jnlp'
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: frontend-test
    image: centos:7
    command:
    - cat
    tty: true
    volumeMounts:
    - mountPath: '/opt/app/shared'
      name: sharedvolume
  - name: backend-test
    image: centos:7
    command:
    - cat
    tty: true
    volumeMounts:
    - mountPath: '/opt/app/shared'
      name: sharedvolume
  volumes:
  - name: sharedvolume
    emptyDir: {}
"""
        }
    }
    stages {
        stage('Test Pipeline Configuration') {
            steps {
                container('frontend-test') {
                    sh 'touch /opt/app/shared/test_file'
                }
                container('backend-test') {
                    sh 'ls /opt/app/shared/test_file ; echo $?'
                }
            }
        }
    }
}

Result:

This solved the problem.

[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘test-pod’ are offline
Agent test-pod-wv12q-brfxj is provisioned from template Kubernetes Pod Template
Agent specification [Kubernetes Pod Template] (test-pod):

Running on test-pod-wv12q-brfxj in /home/jenkins/workspace/pipeline-test
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test Pipeline Configuration)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ touch /opt/app/shared/test_file
[Pipeline] }
[Pipeline] // container
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ ls /opt/app/shared/test_file
/opt/app/shared/test_file
+ echo 0
0
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS


How to track unused resources in AWS? I have been using AWS for a while now. I always have the difficulty tracking AWS resources and how they are interconnected. Obviously, I am using Terraform but still, there is always ad-hoc operations that cut down my visibility.

Since I have been charged multiple times for resources/services that are present but not used by me.

Unused services include resources that are not pointing to other services but present in the AWS environment.

Tools suggestions are also welcome.

There are a few ways to approach deleting unused resources, perhaps tagging items that are part of some trusted or known group, as some have suggested here and then manually monitoring for resources that isn’t tagged appropriately.

You could then extend this to a python script that you run every so often to ID and delete these resources.

However, there are other (potentially less intensive) options.

1) AWS already provide a way for us to monitor our resources, identify unused ones and terminate them with CloudWatch alarms. You can start by configuring some alarming based on some set of checks that you might already be doing manually. Once you are comfortable that the alarming is accurate, you can add a termination step to the chain.

2) Alternatively, our friends at Netflix made this chill tool called JanitorMonkey to monitor and delete unused Instances, EBS Volumes, EBS Snapshots and Autoscaling groups. It is part of the original so-called “Simian Army” open source pack from Netflix and written in Java. Even thought it’s a few years old now, this thing is highly configurable and well worth a look.

Both options here allow you to configure warnings and business rules that fit your needs. Additionally, using CloudWatch might also be a good opportunity to learn more about AWS logs and anomaly identification.


Questions regarding Kubernetes storage I have two questions regarding K8s storage:

Can I set up an NFS server on one of the worker nodes in a bare-metal cluster?
I am running different versions of our application which consists of different components on different namespaces. If I want to use StorageOS do I need to install one StorageOS per namespace?

Thank you.

Yes.  Check out this example for a nice tutorial to get started.
Per the StorageOS documentation you shouldn't even have to worry about this.  Just let StorageOS handle the namespaces for you with one StorageOS install.


error: exportArchive: The operation couldn’t be completed. (IDEDistributionErrorDomain error 3.) I am using Jenkins automation build for iOS.

Previously I have been using xcode 8.1 with no error showing. But i have updated the xcode to 8.3 and I'm facing errors now. I have the scripts as below:

OLD SCRIPT

xcodebuild -exportArchive -archivePath Sample.xcarchive -exportPath Sample -exportFormat ipa -exportProvisioningProfile 'Sampledemo'

xcodebuild: error: invalid option '-exportFormat'

NEW SCRIPT

xcodebuild -exportArchive -exportOptionsPlist exportOptions.plist -archivePath Sample.xcarchive -exportPath Sample  PROVISIONING_PROFILE_SPECIFIER="Sampledemo"

error: exportArchive: The operation couldn’t be completed. (IDEDistributionErrorDomain error 3.)

Error Domain=IDEDistributionErrorDomain Code=3 "(null)" UserInfo={IDEDistributionErrorSigningIdentityToItemToUnderlyingErrorKey={

but I'm facing this issue now. How can I solve the issue ?

https://forums.developer.apple.com/thread/75038

  Hi!

  I think I've got it figured out...

  instead of:

-exportProvisioningProfile "MyProvisioningProfile"

  use:

PROVISIONING_PROFILE_SPECIFIER="MyProvisioningProfile"

  We've got unity building the project every time and it doesn't fill in
  the PROVISIONING_PROFILE or PROVISIONING_PROFILE_SPECIFIER. And it
  looks like it will figure out the alphabet soup that belongs to
  PROVISIONING_PROFILE by looking it up itself provided you have
  downloaded it already.


Jenkins UUIDs ... oh the humanity I have an environment block

environment {
  SUITE_RUN_ID = UUID.randomUUID().toString()
  SMOKE_CMD = "runtests.sh SUITE_RUN_ID=${SUITE_RUN_ID}"
  FAILED_TESTS = "output_rerun_info.rb SUITE_RUN_ID=${SUITE_RUN_ID}"
}

When I use SMOKE_CMD and FAILED_TESTS in their respective stages I get different UUIDs... why?

def suiteRunId = UUID.randomUUID().toString() worked at the top of the Jenkinsfile.

Thanks all for your answers.


Unit Test AWS Glue job in CodePipeline I have an AWS Glue job written in Python that I would like to perform pyunit tests on. However, from reading and experimenting on my own, there seem to be quite a few dependencies on the Glue service, making this task very impractical if not impossible. My only thought would be to separate the non glue dependent functions in the job into a separate module to perform unit testing on.

Is unit testing with AWS Glue possible, and if so, what are the best practices for doing so?

Checkout this solution with codepipeline.  https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/


Why does the jenkins-cli.jar linting functionality not work? The jenkins-cli.jar has been downloaded.

It is able to run.

When one tries to check the syntax of a jenkinsfile by running java -jar ~/Desktop/jenkins-cli.jar -s https://<jenkins_url> -noCertificateCheck declarative-linter < Jenkinsfile it results in:

Sep 27, 2017 10:46:18 AM hudson.cli.CLI _main
INFO: Skipping HTTPS certificate checks altogether. Note that this is not secure at all.
Sep 27, 2017 10:46:34 AM hudson.cli.CLI$5 run
WARNING: null
java.io.IOException: Stream is closed
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3512)
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3486)
        at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
        at hudson.cli.PlainCLIProtocol$EitherSide.send(PlainCLIProtocol.java:188)
        at hudson.cli.PlainCLIProtocol$EitherSide.send(PlainCLIProtocol.java:195)
        at hudson.cli.PlainCLIProtocol$EitherSide$1.write(PlainCLIProtocol.java:215)
        at hudson.cli.CLI$5.run(CLI.java:699)

@Tensibai asked whether there is an authentication issue. Authentication was done, but it is unclear whether it was succesfull or not:

$ java -jar ~/Desktop/jenkins-cli.jar -noCertificateCheck -s https://<jenkins url> login --username <username> --password <password>
Sep 27, 2017 10:57:17 AM hudson.cli.CLI _main
INFO: Skipping HTTPS certificate checks altogether. Note that this is not secure at all.

After upgrading Jenkins the issue was solved. It turned out that the issue was version related.


What is DOCKER_CONTENT_TRUST flag? I am trying to understand this flag DOCKER_CONTENT_TRUST=1 but not able to figure it out. How does it effect pushing images and pulling images, let's say from docker hub?

DOCKER_CONTENT_TRUST=1 this is env variable to enable Docker content trust by default it is in disable mode.

read more about DCT in official docs DCT docs

For specific  build/push/pull you can use as docker command line flag as well

--disable-content-trust=false or --disable-content-trust=true


Identifying a benchmark to compare overall performance of VM virtualization vs hardware Which performance index is most suitable to find out real performance of your VM limited by sharing and quotas?

For example, if you have a physical server with 4 cores and 16 GB RAM, its performance limit might be higher than if you have same on AWS or in some VMWare product.

How to compare to find out the gap? Is there an overall index or better to use some reference applications' performance as benchmark? (tough, though, if you still going to develop that application)

You should do a comparison of commercial virtualization products on the same hardware resources, OR benchmarking commercial virtualization products individually, there are ways to do it, you can even script it.

For example, VMmark is a free tool used to measure the performance and scalability of virtualization platforms VMmark

Since your question is very generic, you should clarify real performance, overall performance. It might mean "real performance in virtualization".


Terraform ignore_changes block on JSON-encoded attribute I'm currently trying to get CloudWatch logging and DMS configuration working smoothly in my module by ignoring certain attributes on replication_task_settings. Right now, I have a DMS replication task that looks like the following:
resource "aws_dms_replication_task" "main" {
  migration_type            = "full-load-and-cdc"
  replication_instance_arn  = var.replication_instance_arn
  replication_task_id       = "${var.environment}-${var.customer}-dms-replication-task-ongoing-main-${var.engine}"
  source_endpoint_arn       = var.source_endpoint_arn
  target_endpoint_arn       = var.target_endpoint_arn
  replication_task_settings = jsonencode(local.task_settings)
  table_mappings            = jsonencode(local.table_mappings)
  tags                      = merge({}, local.additional_labels)

  lifecycle {
    ignore_changes = [
      replication_task_settings
    ]
  }
}

With this configuration, I'm ignoring all future changes to replication_task_settings. What I would like to do is only ignore changes to only Logging.CloudWatchLogGroup and Logging.CloudWatchLogStream (not all of the task settings) since DMS creates these for you and you can't change them. The problem is that the replication _task_settings is a JSON-encoded attribute. If I try to do ignore_changes = [replication_task_settings["Logging"]["CloudWatchLogStream"]], I get this error:
Error: Invalid index

  on modules/repl-ongoing/main.tf line 250, in resource "aws_dms_replication_task" "main":
 250:       replication_task_settings["Logging"]["CloudWatchLogStream"]

This value does not have any indices.

If I try to decode the JSON by doing: ignore_changes = [jsondecode(replication_task_settings)["Logging"]["CloudWatchLogStream"]], I get the following error:
Error: Invalid expression

  on modules/repl-ongoing/main.tf line 250, in resource "aws_dms_replication_task" "main":
 250:       jsondecode(replication_task_settings)["Logging"]["CloudWatchLogStream"]

A single static variable reference is required: only attribute access and
indexing with constant keys. No calculations, function calls, template
expressions, etc are allowed here.

Is there a way I can ignore changes to specific properties of JSON-encoded attributes? Or is there another workaround with DMS tasks that I have to go with?

Due to how that resource type is designed, from Terraform's standpoint replication_task_settings is just an opaque string and so Terraform can either detect that the string has changed (the default) or ignore changes to the string as a whole.
There is no way to ignore changes to only part of a string, because Terraform implements ignore_changes by taking the previous value of that argument instead of the new value. Terraform doesn't and can't know how to construct an new string that combines some characters from the old string and some characters from the new string, because it has no awareness of the meaning of that string.


docker volumes on cifs share cause permission disaster I have a small vm running docker it only has a 20gig SSD which is more than enough for the OS + SWAP + Docker and the container images. But now I want to run a nextcloud docker image on that host. Because of the lack of space I mounted a cifs share to /data and started docker linking nextclouds data directory to the share.

 docker run -v /data/docker-volumes/nextcloud/data/:/var/www/html/ -p 8080:80 nextcloud

The files are created but then docker tries to change the ownership of the files to www-data which does not work because the user www-data does not exist on the target system (the SMB Server). Which fails and terminates the docker instance.

rsync: chown "/var/www/html/data" failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]

There is no way to create the user www-data on the target system. Is there any way to get around this?

The problem can be solved by using mount options that force the application of the correct user and group eventhough these attributes can't really be set on the target system its sufficient to get around the docker related problem.

Also a reasonable set of privileges needs to be defined that is used for new files and folders.

mount -t cifs -o username=user,password=pass, \               # provide creds
  uid=www-data,forceuid,gid=root,forcegid, \                  # force gid and uid
  file_mode=744,dir_mode=744 //host/share /local/mountpoint   # set permissions for new files and directories


Why are unnamed volumes created when `docker-compose up` is run and do these double after issuing `docker-compose down && docker-compose up`? When starting a docker compose application, it creates named volume the first time around and then additional volumes each time it is started. Is this the normal behavior in that it creates incremental volumes or is my docker-compose file broken? In addition, after running docker volume prune, it removed two of the unnamed volumes

This is the output after starting it twice:

DRIVER              VOLUME NAME
local               2b33c81e6e955ff36061f4120b7181c7efc7aebded2f87eaa7290027f1e7a725
local               74301eadc75018d6f7da76aec44bcf743e03261492cfcc33211941684a570030
local               c42b1e2c17342c52b4b9e90f4d4b7fa24ccb30de5479e65cf4366e2373dfb0bf
local               dcfa63dbf99362fc92fe3b981f50abebfe1bba03063d4dd9dd790f4c058817cf
local               dockerinfluxdbgrafana_grafana-data
local               dockerinfluxdbgrafana_influxdb-data

And for reference my docker-compose file

version: "3"
services:
  influxdb:
    image: influxdb:latest
    container_name: influxdb
    ports:
      - "8086:8086"
    env_file:
      - 'env.influxdb'
    networks:
      - backend
    volumes:
      - influxdb-data:/var/lib/influxdb

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    env_file:
      - 'env.grafana'
      - 'secrets.grafana'
    networks:
      - backend
    volumes:
      - grafana-data:/var/lib/grafana

networks:
  backend:

volumes:
  influxdb-data:
  grafana-data:

Steps to reproduce

docker-compose up -d && docker-compose down && docker-compose up -d && docker-compose down && docker-compose up -d && docker-compose down

Research indicated that the anonymous volumes were created by influxdb and grafana/grafana.

  Anonymous volumes are not given an explicit name when they are first
  mounted into a container, so Docker gives them a random name that is
  guaranteed to be unique within a given Docker host. Besides the name,
  named and anonymous volumes behave in the same ways.

Results

version: "3"
services:
  influxdb:
    image: influxdb:latest
    container_name: influxdb
    ports:
      - "8086:8086"
    #env_file:
    #  - 'env.influxdb'
    networks:
      - backend
    volumes:
      - influxdb-data:/var/lib/influxdb

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    #env_file:
    #  - 'env.grafana'
    #  - 'secrets.grafana'
    networks:
      - backend
    volumes:
      - grafana-data:/var/lib/grafana

networks:
  backend:

volumes:
  influxdb-data:
  grafana-data:

results in:

DRIVER              VOLUME NAME
local               604a07040367512b09c618c6dcc71a7f55390c9c23de6ab08be7466414ed62da
local               7f4b630073b31b6e772d3edef6da81b48643525edfc34281ea13fbd6b86ec270
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data

and everytime when docker-compose down and subsequenlty up is run the number of anonymous volumes doubles.

What will happen if nginx will be pulled instead of influxdb and grafana?

DRIVER              VOLUME NAME
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data

and after docker-compose down && docker-compose up -d?

DRIVER              VOLUME NAME
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data

It looks like that certain images created the additional anonymous volumes. Let's replace influxdb with nginx and use grafana.

DRIVER              VOLUME NAME
local               15b80416ab06abb629d9f634a0feff08f7c560f31d614b9b430855c16cdb75c7
local               205a6f19cbf992c95b2e3be9f2fb1ca9ecec35fce550d0b7a4b9f32b0ef163b1
local               474108f5b7b14fba92a3e5a980f3bf851388b2ee25d7417df5c42d9f176e084b
local               5830a31a470ec8a42ddae7a37bb50487f3f36360318b2f9f5301b338507782b4
local               9f00868a2fec0cfc0d34dc12d0879d39487a13128863722f400ad4c47df2d340
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data
local               f47b1b7bbec8e50b32a7c39704c7c218165b284298d852313fa24bc7cbe6acc5

Everytime docker compose was run again, three anonymous volumes were created by the grafana/grafana docker image. Let's replace influxdb with nginx and revert the grafana to nginx:

DRIVER              VOLUME NAME
local               devopsstackexchange_grafana-data
local               devopsstackexchange_influxdb-data

and it remains two if restarted.

It looks like that grafana is causing the issue.

Why are three new anonymous volumes created everytime grafana/grafana is restarted?

The grafana/grafana dockerfile indicates that three anonymous volumes will be created:

VOLUME ["/var/lib/grafana", "/var/log/grafana", "/etc/grafana"]

https://docs.docker.com/engine/reference/builder/#volume

FROM ubuntu
RUN mkdir /myvol
RUN echo "hello world" > /myvol/greeting
VOLUME /myvol

  This Dockerfile results in an image that causes docker run, to create
  a new mount point at /myvol and copy the greeting file into the newly
  created volume.

docker volume inspect <volume name, e.g. 34cfafd4603dbc7e71a83e2520f978c8307b084143b3192de65a1995dc1d2f86> returned grafana data when the path that was returned was checked of two out of three anonymous volumes:

sudo ls /var/lib/docker/volumes/a0ecd00df8fc68ef36e777c7bf9ec5a496ee30e313b86889487501a53fa2e28e/_data
grafana.ini  ldap.toml

and

sudo ls /var/lib/docker/volumes/34cfafd4603dbc7e71a83e2520f978c8307b084143b3192de65a1995dc1d2f86/_data
grafana.db  plugins


403 forbidden error on calling jenkins post api from python script I am trying to build a python script that will build jobs externally but when I am executing it, throws 403 forbidden error for build jobs call other get APIs are working fine. Below is the python snippet.

j=jenkins.Jenkins('http://localhost:8080/login?from=%2F', 'admin', 'admin@123' )

j.get_jobs();

print(j.jobs_count());

print(j.get_jobs());

j.build_job('demo1');

Jenkins details: I am able to create & build jobs from UI successfully. My Jenkins version is 2.237.

Other settings.

logged in user can do anything. Csrf protection is disabled.
On the other hand my friend having the same script and version of Jenkins is able to build job.

As recommended by Jenkins (>2.2xx) you may benefit by creating an API token for the user, I did so, to trigger Jenkins job in my local Jenkins server from NODEJS code.
Please refer below for further clarification:

https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/#shell-with-curl
https://www.jenkins.io/doc/book/using/remote-access-api/#RemoteaccessAPI-CSRFProtection

And for details: https://www.jenkins.io/doc/book/managing/security/


What is the future role of tester? If the new paradigm is "You build it, you run it" (Werner Voegels, Amazon CTO) which obviously puts much more responsibility - and pressure - on software engineers, what does this change introduce to the task of the test team?

IMHO that depends on what the role of the tester was before such transformation. BTW, I believe my answer applies to the DevOps transformation in general, not only to the You build it, you run it paradigm.

If the role was a drone job - mindlessly executing tests - it's bound to go away, automation will eat such jobs.

If the role included writing the test plans and specifications they could continue to do that or even morph into writing the corresponding automation scripts, side-by-side with the developers. Or standalone - in some environments it is mandatory to use a completely separate methodology/infrastructure/personnel for testing than the one(s) used for development. In many organisations testers are software engineers, just as developers.

If the role was executing intentionally unreliable/surprise tests, specifically requiring human traits (emotion, subjectivity, inspiration, reflexes, intuition, etc) to complement automated testing - that also won't change.


stop pulling a docker image from the public docker-hub registry Is there any way to stop pulling a docker image from the public docker-hub registry while building image locally.
Note: We are maintaining a private registry that contains all the images. Only and only docker-engine should search for an image from the private registry and not from the public registry.

You cannot change the default registry (see this). You have to specify which registry to use like this:

docker pull myprivatereg:port/library/image

Or, in a Dockerfile:

FROM myprivatereg:port/library/image

(This question was also asked in StackOverlow).


Kops: How to delete the multiple SSHPublicKey SSHPublicKey    admin   2b:xx:
SSHPublicKey    admin   2c:yy

How to remove one ssh key from the cluster ?
when tried to delete via normal way,

kops delete secret --name cluster-1.dev.transein.com sshpublickey admin

found multiple matching secrets; specify the id of the key

How to remove one , or even the entire sshkey

kops delete secret --name cluster-1.dev.transein.com sshpublickey admin 2c:yy


Deploying a set of cloud instances based on a list of parameter differences Sorry for the general and vague nature of my question.  Let me describe with some more detail what I would like to do (or what I think I would like to do).

I have a shell script for ubuntu 16.04 that sets up my instance how I want it.  But it needs to use two different variable inputs for each different instance.  For example, if across all the instances that I wanted to set up, I wanted to put in different timezone, I could start with a list of all the timezones, and iteratively deploy a machine for each timezone.  Maybe the solution I look for would drop a textfile with the timezone I wanted on each instance. That would be enough for me to read the file and set it up.  I just need a solution that will allow me to create any number of cloud instances of ubuntu 16.04 and deliver seperate textfiles with these timezones.  They are all identical except the file with the timezone.

I'd like to be able to create all these instances simultaneously from the command line or through some R package for an API interface. Digital Ocean, AWS, or even some local option if I should be considering that.

Is this a job for docker or vagrant?  What is the tool for this kind of deployment? Maybe a script to loop through creating instances with these?

This is why configuration management software exists.
Example for Ansible and AWS:

- name: Create a sandbox instance
  hosts: localhost
  gather_facts: False
  vars:
    timezones_path: /some/path/timezones.list
    key_name: my_keypair
    instance_type: m1.small
    security_group: my_securitygroup
    vpc_subnet: my_vpc_id
    image: my_ami_id
    region: us-east-1
  tasks:
    - name: Load timezones list
      shell: "echo {{ item }}"
      with_lines: "cat {{ timezones_path|quote }}"
      register: timezones

    - name: Launch instances
      ec2:
         key_name: "{{ keypair }}"
         group: "{{ security_group }}"
         instance_type: "{{ instance_type }}"
         image: "{{ image }}"
         wait: true
         region: "{{ region }}"
         vpc_subnet_id: "{{ vpc_subnet }}"
         assign_public_ip: yes
         count: "{{ timezones|length }}"
      register: ec2

    - name: Add new instance to host group
      add_host:
        hostname: "{{ item.1.public_ip }}"
        timezone: "{{ item.0 }}"
        groupname: launched
      with_together:
        "{{ timezones }}"
        "{{ ec2.instances }}"

    -name: Tag instances with timezones
      ec2_tag:
        region: "{{ region }}"
        resource: "{{ item.id }}"
        tags:
          Timezone: "{{ item.timezone }}"
      with_items: "{{ groups['launched'] }}"

    - name: Wait for SSH to come up
      wait_for:
        host: "{{ item.public_dns_name }}"
        port: 22
        delay: 60
        timeout: 320
        state: started
      with_items: "{{ ec2.instances }}"

- name: Configure instance(s)
  hosts: launched
  become: True
  gather_facts: True
  tasks:
    - name: Run deploy script
      shell: "/path/to/script.sh --tz {{timezone}}"

Path to script in last line should be for host which runs Ansible, not remote.


Does Prometheus expose the Horizontal Pod Autoscaler's "Current CPU Utilization" as shown in the Kubernetes dashboard? In the Kubernetes dashboard, I can see for a HPA the following information:

Min Replicas:
3
Max Replicas:
11
Target CPU Utilization:
80%
Status
Current Replicas:
3
Desired Replicas:
3
Current CPU Utilization:
10%
Last Scaled:
5 days

However, I can't figure out how the 10% CPU utilization is calculated?

And if this value exposed via Prometheus?

As we can see from kube-state-metrics/docs/horizontalpodautoscaler-metrics.md

We just need to have kube-state-metrics along with HPA to start getting metrics in prometheus.


Multi-Stage Azure Pipeline with a path trigger - first stage always runs I have a multi-stage pipeline for my terraform code. This is my first pipeline so I am sure I'm missing something but there appears to be some inconsistent behaviour that is rather frustrating.
The pipeline trigger is defined as follows:
trigger:
  branches:
    include:
      - master
  paths:
    include:
      - terraform/*

My repo has the following folder structure:
IaC
 |
 +-- terraform
 |    |
 |    +-- main.tf
 |
 |
 +-- azure-pipeline.yml
 +-- readme.md

My two stages are defined as follows (I cut out the job steps, its just tarraform commands):
stages:
  - stage: plan
    displayName: 'terraform plan'
    condition: eq(variables['Build.Reason'], 'PullRequest')
    ...
  - stage: apply
    displayName: 'terraform apply'
    condition: ne(variables['Build.Reason'], 'PullRequest')
    ...

Whenever I update code in the terraform directory and create a PR, the pipeline works exactly as expected.  It does the terraform plan for the reviewer to see and once the PR is approved, the second stage fires and it does a terraform apply.
The issue I am having is whenever I update a file outside the terraform directory and create a PR, the first stage still runs! Oddly, once the PR is approved, the second stage does not.
Are my stage conditions to simple? Does a multi-stage pipeline always have to do something?

Your pipeline runs for pull requests not because of the trigger, but because of a build validation policy you've set for one or more target branches ("whenever I create a PR to this branch, start this pipeline"). It's the build validation, as part of a branch's policies: https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops#build-validation
You probably didn't specify a path filter in the build validation settings. Here's an example of three different build validations in a single branch policy, all with different path filters:

In short: The path trigger in your YAML has no effect on how a pipeline is triggered for a PR. The trigger you've specified is one that triggers on commits to the master branch, if the commit includes files in the terraform directory.


Can I move SSL certificate files from one server to another for the same domain? I have a website running at https://domain.com and IP a.a.a.a, with an SSL cert issued by GoDaddy. The server is deployed with Nginx. Everything is working fine as it is.
However I'm trying to deploy the same website with Docker in a new server with IP b.b.b.b, so I did the following:

Pull nginx image from Docker
Copy the domain.com.chained.crt and domain.com.key files from server a.a.a.a to server b.b.b.b
In the Docker file, I copy the cert files to the image
When I run the container and check nginx with nginx -t, it seems to be working fine.
Ports 443 and 80 are open for external requests
I check that the website is running by accessing to b.b.b.b:80 from an external client
Configure the DNS to resolve to b.b.b.b instead of a.a.a.a
Check that http://domain.com is working

Up to that point everything is working fine. But when I try to access https://domain.com I get the following errors:

With firefox: ERR_SSL_PROTOCOL_ERROR
With chrome: SSL_ERROR_RX_RECORD_TOO_LONG
The site https://www.sslshopper.com/ssl-checker.html tells me this: No SSL certificates were found on domain.com. Make sure that the name resolves to the correct server and that the SSL port (default is 443) is open on your server's firewall.

The Nginx error logs don't tell me much, although I have them in debug mode.
My main question is: Can I copy the cert files from one server to another or do I have to generate them again?
Edit:
This is the docker-compose.yml
sevices:
  nginx:
    build: ./nginx
    image: nginx
    ports:
      - "80:80"
      - "443:443"
  redis:
    image: redis
    ports:
      - "6379:6379"
  mongo:
    image: mongo
    ports:
      - "27017:27017"
    volumes:
      - /var/mongodata:/data/db

Dockerfile
FROM custom:latest

COPY nginx.conf /etc/nginx/nginx.conf
COPY /ssl/domain.com.key /etc/nginx/ssl/domain.com.key
COPY /ssl/domain.com.chained.crt /etc/nginx/ssl/domain.com.chained.crt

RUN apt-get -y install nodejs libssl1.0-dev  nodejs-dev node-gyp npm
[...]
CMD ["nginx", "-g", "daemon off;"]

nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 768;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log debug;

    ##
    # Gzip Settings
    ##

    gzip on;

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    #include /etc/nginx/sites-enabled/*;

    server {
        listen 80;
        listen [::]:80;
        listen 443;

        server_name www.domain.com;
        ssl_certificate /etc/nginx/ssl/domain.com.chained.crt;
        ssl_certificate_key /etc/nginx/ssl/domain.com.key;

        location / {
            root /domain.com/;
            try_files $uri$args $uri$args/ /index.html;
        }
    }
}

Your listen 443 need to have ssl parameter to tell Nginx that all connections accepted on this port should work in SSL mode.
See example at https://nginx.org/en/docs/http/configuring_https_servers.html#single_http_https_server
Shown here for convenience:
server {
    listen              80;
    listen              443 ssl;
    server_name         www.example.com;
    ssl_certificate     www.example.com.crt;
    ssl_certificate_key www.example.com.key;
    ...
}


How to enable the npm cache in a Jenkins build slave that runs in docker? I have a Docker image, let's call it frontend.image, that I use for a Jenkins build slave. The Jenkins Docker plugin will spin up a container from this image and build artifacts inside the container. This all works great. In this case, the frontend.image is used to build an AngularJs app. Part of building this Angular app is to install npm packages the app requires.

This process, npm install, seems to take a long time, 3 minutes it seems, npm always installs every package every time.

So I added a volume for my slave, it is a host mounted volume, the Docker plugin will use this volume every time it runs the frontend container:

The user that executes the command npm install is jenkins. npm keeps a cache which you can find with command npm config get cache which outputs /home/jenkins/.npm

That is why I have the host volume /slaves/volumes/tsl.frontend:/home/jenkins mounted to my web container slave.

I build my Angular app using a Jenkins project, builds no problem, many npm packages are installed. If ssh into my Docker host and run cmd ls /slaves/volumes/tsl.frontend I see lots of npm packages. This means my host volume mount for the slave worked.

Now I build the Jenkins project again, npm installs every single package again, even though the Docker slave build container is using the volume host mount. I can even confirm by bashing into the slave container with cmd docker exec -it <some_clever_random_container_id> bash then cmd su jenkins then cmd npm cache ls which lists many npm packages that are cached.

So, even with my host mount volume, which has permissions chmod 777 by the way so there is no permissions issues, I cannot get npm install to use the cache.

In my Jenkins build, which spins up the Docker slave container, the first cmd I run is npm cache ls and many packages are listed, doesn't this mean my host volume is working as expected and the npm cache index has integrity aka not corrupted?

I have tried the regular npm install cmd, which, when I run on my localhost machine installs all the packages the first time and almost no packages the next time. And also the npm cache "hack" npm --cache-min 9999999 install, taken from this SO answer as well as cmd npm --skip-installed --cache-min 9999999 install

A related question was posted on StackOverflow.

I finally solved this by using Docker image layer caching for the npm install, following this answer

This means I moved the npm install out of the Docker slave image and into the actually frontend image, here is my final Docker file that truly caches the npm install in between builds if package.config has no changes:

FROM centos:7
MAINTAINER Brian Ogden

# Not currently being used but may come in handy
ARG ENVIRONMENT
ENV NODE_VERSION 6.11.1

RUN yum -y update && \
    yum clean all && \
    yum -y install http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm \
    yum -y makecache && \
    yum -y install nginx-1.12.0 wget

# Cleanup some default NGINX configuration files we don’t need
RUN rm /etc/nginx/conf.d/default.conf

#############################################
# NodeJs Install
#############################################

#Download NodeJs package
RUN wget -q -O - https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz \
    | tar --strip-components=1 -xzf - -C /usr/local

# https://stackoverflow.com/a/35774741/1258525
# use changes to package.json to force Docker not to use the cache
# when we change our application's nodejs dependencies:
COPY ./package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir /app && cp -a /tmp/node_modules /app/

WORKDIR /app
COPY . /app

RUN npm run build-$ENVIRONMENT

RUN cd /app && cp -a dist/* /usr/share/nginx/html
COPY ./docker/conf/frontend.conf /etc/nginx/conf.d/frontend.conf
COPY ./docker/conf/nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

CMD ["nginx"]


When would I choose IPTables over an Ambassador pattern for port forwarding? In an environment of Secured Virtual Private Clouds, in a Cluster of Docker Containers, we have to setup some routes.

We can do that with the Ambassador pattern, which is simple and easy to maintain. (Docker does have some usage of IPTables under the hood - but from what I can see the Ambassador pattern uses socat, and not IPTables to achieve its forwarding.

We can setup NAT rules with IPTables to achieve a similar goal.

My question is: When would I choose IPTables over an Ambassador pattern for port forwarding?

iptables will probably scale better than socat because socat runs completely in user space.  There's a common impression that iptables is hard to deal with, but I find it easier that most routers to fiddle with.  There are a number of iptables wrappers available to make it easier to handle.

If you're willing to accept the performance hit of running in user space then consider haproxy over socat because it is intended to run over longer periods and provides statistics, failover, SSL termination, and lots of other conveniences.


Six repositories as a single deployment Docker+CircleCI I need some help and be pointed in the correct direction.

I have 6 git repositories that together combine a bigger application (3 Node.js API, 2 React web apps and one repository for Ansible and templates for config files for the other repos).

Currently we are building it only with Ansible and CircleCI but the current build process is not that great and I think of moving to Docker, docker-compose and CircleCI 2.0.

The thing is: how should I dockerize it?

All the databases are remote. How should I approach nginx?

I thought about writing Dockerfiles for each application and then using Ansible to connect to host, install Docker, pull sources and start building containers from Dockerfiles.

Nginx should be configured by Ansible on host or can it be made into a Docker container somehow?

With docker compose (or docker swarm if you want to distribute the containers) create a container for each Git repository. This should map well to docker’s philosophy of each container being used for a single purpose.

As far as examples go, I can recommend you have a look at veleda (shameless self-promotion) which combines 3 databases, a flask app, a ruby app (Grafana), Jekyll and nginx. It also includes unit tests that are integrated into Travis-ci and should be transferable to circle-ci.

Regarding the nginx aspect, 030 already mentioned that there are finished images to make your life easier. Even ones that include automated Let’s Encrypt services.

Ansible can be integrated well into the mix. If you used it to install dependencies and network configurations, this is now handled by docker. For SSH-based tasks, ansible remains useful. Check out this comparison of tools and their scope’s of responsibility.


How to ensure EC2's OS/application is loaded? Recently I was playing around a bit with AWS as a part of a course that I am doing. I came across a situation that to be honest is keeping be in vilo.

I made a web application which connects to a backend that basically launches a EC2 t2.micro instance on AWS. From time to time, I pull the status of the machine to check whether it is on running state or note. The AMI I am using it is a configured wordpress from the marketplace.

As soon as the instance gets to the running state, I show a message indicating the server is up and running and provide the user with the instance's public ip address, the problem is that, it is not after a few minutes later when indeed the IP is resolved/accessible. The security group attached to the instance has the inbound and outbound traffic configured to allow TCP traffic on port 80, 22 and 443.

My question is, the running state simply means the instance is ready to work but, as far as I know, it does not mean OS has booted nor the X application (in this case wordpress and its configured server) are ready to or finished the bootstrapping. How can I check that indeed the application/OS has booted is I can ensure that when the user is trying to access the IP it will not timeout.

My first attempt was to ping the instance bu to be honest I don't think it's a reliable request since, I'd also need to enable the ICMP protocol but at the end, it's working in another internet layer.

What you are talking about is known as a health check. Typically these are handled by load balancers, to decide whether instances are "healthy", and thus can serve (usually HTTP) traffic. The load balancers will poll and endpoint that you define to decide whether the instances are healthy, usually by seeing if the HTTP response code is some known-good one, like 200.

One solution therefore, could be to front your application with a load balancer, and enable health checks on the relevant urls. This will check whether your application is serving traffic properly, when polled by the load balancer. While this adds a level of complexity (and cost) to the app, it is also quite easy to do.

Of course, you could mimic this functionality without an AWS load balancer, by having your app poll the url that the app is supposed to respond on, by using the DNS or IP name returned by AWS after the instance becomes "running".

Another option would be to set up an actual monitoring system to check whether the application is serving content. A very simple and free tool that would do the job is e.g. Freshping.

You could also write an Inspec Compliance profile to describe a known good state of the machine. Inspec can connect to the machine via SSH, which you specify as being open. This may sound like overkill, and quite similar to the ICMP test you considered before, but it's nevertheless an option :)


"Software Development" which environment is it: Manufacturing or Projects? Is it Manufacturing or Projects?

Definitions I am using for these:

  Project

  Planned set of interrelated tasks to be executed over a fixed period and within certain cost and other limitations.

and

  Manufacturing

  The process of converting raw materials, components, or parts into finished goods that meet a customer's
  expectations or specifications. Manufacturing commonly employs a
  man-machine setup with division of labor in a large scale production.

I can suggest that in software development manufacturing translates to:

"raw materials" = requirements and specifications
"parts" = software libraries
"finished goods" = service/software deployed and serving customers
"man-machine setup" = configuration of environments
"division of labor" = dev and ops and qa, etc...
"large scale production" = lots and lots and lots of code

When I write "Software Development" I actually mean the whole cycle that includes lead-time from the point where some customer need is discovered, to the point of customer value created and delivered.

According to Wikipedia

  Manufacturing is the production of merchandise for use or sale using
  labour and machines, tools, chemical and biological processing, or
  formulation.

and

  a project is an individual or collaborative enterprise, possibly
  involving research or design, that is carefully planned, usually by a
  project team, to achieve a particular aim.

Based on the definitions Software Development could both be seen as Manufacturing and a Project.

It could be seen as manufacturing from a deliverables perspective as Development activities mean that that the software is created, but does not guarantee quality. Therefore QA is required and to get it online Operations tasks are needed as well. In summary, Software Development could be seen as manufacturing as multiple specialisms are required to make the feature available for a customer.

It could also be compared with a project as the Dev team itself does R&D and creates a (software) product.

From a DevOps perspective I would advocate to compare Software Development with manufacturing rather than project as DevOps is about let work multiple specialisms together and not about creating silos like project.


Cross account access from ec2 to other accounts Is it possible to run aws cli from ec2 to create some resources in another account without using aws_secret_key and aws_access_key?

Supposing the scenario with two accounts A & B the explanatory steps should be:

In account A, I created a role (e.g RoleForB) to trust account B, and attach to the before created role a IAM policy to allow it to perform some read operations in account A. e.g ReadOnlyAccess
In account B, I created a role (e.g AssumeRoleInA) and attach a policy to allow it to assume the role that is created in account A.
In account B Associate to your EC2 instance ec2-profile the IAM role (AssumeRoleInA) created in step 2.
In account B login into this EC2 instance to assume the role in Account A using the command aws sts assume-role --role-arn "arn:aws:iam::Account_A_ID:role/RoleForB" --role-session-name "EC2FromB".
In account B EC2 terminal when the command is step 4. finished, you can see the access key ID, secret access key, and session token from wherever you've routed it, in our case stdout either manually or by using a script. You can then assign these values to environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)

So Let’s check the configurations mentioned above step by step but with some mode detail:

As before presented in account A, it builds the trust to account B by creating the role named RoleForB and attaching ReadOnlyAccess permission to it.

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Principal": {"AWS": "arn:aws:iam::Account_B_ID:root"},
        "Action": "sts:AssumeRole"
    }
}

In account B, create a role named AssumeRoleInA then attach the corresponding policy to allow it to assume the role named RoleForB in account A.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": [
        "arn:aws:iam::Account_A_ID:role/RoleForB"
      ]
    }
  ]
}

In account B, create a new EC2 instance (if it does not exists yet), and associate it's ec2-profile with the IAM role named AssumeRoleInA.

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Principal": {"Service": "ec2.amazonaws.com"},
        "Action": "sts:AssumeRole"
    }
}

In account B login into this EC2 instance to assume the role in Account A using the command:

aws sts assume-role --role-arn "arn:aws:iam::Account_A_ID:role/RoleForB" --role-session-name "EC2FromB"`

eg:

jenkins@bb-jenkins-vault:~$ aws sts assume-role --role-arn arn:aws:iam::521111111144:role/DeployMaster --role-session-name "project-dev-jenkins-deploy"
{
    "AssumedRoleUser": {
        "AssumedRoleId": "AROAJBXGEHOQBXGEHOQ:project-dev-jenkins-deploy",
        "Arn": "arn:aws:sts::521111111144:assumed-role/DeployMaster/project-dev-jenkins-deploy"
    },
    "Credentials": {
        "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "SessionToken": "FQoGZXIvYXCUm8iG6/zLdQ7foognvCDpxKP7cRJiZgc...CUm8iG6/zLdQ7foognvCDpxKP7c+OQF",
        "Expiration": "2019-03-29T15:41:02Z",
        "AccessKeyId": "AKIAI44QH8DHBEXAMPLE"
    }
}

In account B EC2 terminal when the command is step 4. finished, you can see the access key ID, secret access key, and session token from wherever you've routed it, in our case stdout either manually or by using a script. You can then assign these values to environment variables

$ export AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
$ export AWS_SESSION_TOKEN=FQoGZXIvYXCUm8iG6/zLdQ...<remainder of security token>
$ aws ec2 describe-instances --region us-east-1

credit: https://stackoverflow.com/a/55420220/6657158


Having trouble with building Docker image I'm trying to build a docker image for a software using docker build -t cytools .
Here is the output that I'm confused about:
[+] Building 0.0s (1/2)
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 2B                                         0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount346931491/Dockerfile: no such file or directory

Does anyone know what this means? I am new to all this, so sorry if I didn't use the correct terminology or give the full picture!

Looks like the Dockerfile is not in the path, '.' means the current directory you are in.


Do you use existing automation code (cookbooks/playbooks/dockerfiles/etc) or you write your own from scratch? I noticed that many cookbooks/playbooks/... are not of particularly high quality, and even if they are they rarely cover all cases. So it should be expanded with my custom logic everytime, while 90% of the *book remains unused (+ adds complexity/reduces ease of modification/lacks unification between *books/...).

In particular, configuration management most of the time is "high-level enough"  to not write too many boilerplate even if you implement something from scratch. The only exception might be additional "pure modules" for specific use cases if DSL does not provide enough functionality.

I understand this is pretty subjective, but I'm truly surprised I can't find anything related to this (pretty basic) topic. What is your experience, can we say there is a best practice for when to implement your solution from "scratch" or use the existing infrastructure/deployment automation code?

My limited experience is indeed that often, you can get a first solution very quickly with pre-made automation code; unfortunately, often there either appear unexpected problems during "first contact", or later during use, especially when you wish to expand the configuration for unusual cases.

So, indeed, I do tend to approach this with an open mind. I will certainly start out with something pre-made, but sometimes my gut feeling tells me to take a look under the hood, and then I'll decide rather spontaneously if I better do it myself, or whether it is a good idea to roll with it...


Jenkins Slaves cannot connect to the Master Jenkins I switched the Jenkins to using https on port 8080 . After a restart of the Jenkins,the slaves cannot reconnect.
I changed only Jenkins location from Configure System -> Jenkins Location -> Jenkins URL to https://localhost:8080
Know someone who can answer?

It depends on how you connect you slaves via ssh or jnlp,
in any case: check if you are able to connect that slave and try telnet it.


Will google cloud take money from me in this situation? I registered google cloud account and want to use Text-To-Speech API on my own server with github.com/googleapis/google-cloud-php library.

On API page in block "Pricing" written that using less than 1 millon chracters/month is free. But lower is a note
"Additional charges may also apply from infrastructure you use to call the API"

Question:
This note applies to users google cloud VPS, right? If I will use API how I described (send requests from my own server) first million of characters will be absolutely free? Google wont take a cent?

Yes, they mean the Compute Engine. So you pay Google only for API, not for server.


Is it an antipattern to avoid API Gateway for certain tasks? I'm creating a platform architecture for my project and I'd like to use an API Gateway of some sort for handling most requests to a Kubernetes cluster (with a few microservices).

There is, however, one task that relies a lot on performance and sheer number of connections (think IoT metrics kind of traffic, a lot of connections sending data often, most likely gRPC). I'd always hit a single service in the cluster. It's also different because it will come always from a single physical device (IoT hub, if we're using that analogy) unlike the rest (being web browser, mobile app, 3rd party access, etc).

Since I'd like to offload the API Gateway from that kind of traffic to provide stability and performance for remaining traffic, would it be an antipattern to have a direct connection for this case? Or maybe there's another solution for it?

You could have a separate API gateway for that kind of traffic.

Check out the "Variation: Backends for frontends" section of the following article:

https://microservices.io/patterns/apigateway.html


How do I autostart Jenkins agent machines? To save money, I would like a large number of agent machines to be stopped and started up only when needed. Better yet, a single  VM should be copied and launched on demand.

I am using Google Cloud Platform.

I see the  Kubernetes plugin, used in Jenkins X, as well as the  VirtualBox Plugin, but I get the sense that Jenkins might not be designed for this.

What is the standard way, if any, to do this?

We use AWS's EC2 Plugin to spin up spot instances in AWS for our Jenkins master based on a single AMI, exactly as you describe, so Jenkins does support this behavior.

From a quick bit of searching Google Cloud seems to offer a similar solution;

  Spotinst’s Jenkins Plugin helps you to do more with your Jenkins setup by allowing you to automatically configure and scale a designated Google Compute Engine Preemptible VM as Slaves up and down depending on the number jobs to be completed.


Host credential manager and Docker runners Imagine you have a GitLab Docker runner which you would like to pass credentials from a credential store. That is the CI/CD variable just designates the credentials profile name.
Is there a way to connect Docker containers to host's credential store?

You need to provide more information about the technology stack.
In a nutshell, you would mix the responsibilities of the host, taking care for the gitlab runnner + credential store is not a good design practice. If the host goes down, you are loosing 2 essential services, instead of just one. Most likely you would have other consumers of the store. Also this is breaking the single responsibility principle.
I could tell you from own experience is that we use 'AWS Parameter store' for that  ( H/A and detached service ). The secrets are consumed by a helm chart - https://github.com/cmattoon/aws-ssm and injected into the cluster as configurations. After that each interested party could use whatever it wants.


Is there a tool that makes it possible to validate Jenkinsfiles online? At the moment I am editing a Jenkinsfile and then let it run unless Jenkins reports a issue. This approach costs a lot of time. I prefer to validate the syntax before committing the Jenkinsfile. Is there a tool that solves this issue?

In gitlab there is a URI, i.e. /ci/lint that makes it possible to submit a gitlab file, click on the check button and then the UI will indicate whether the syntax is correct or not.

Here is some documentation on the Jenkins pipeline linter and its commands. Do you need to validate before a commit? If not, it would be really trivial to run the linting command prior to your pipeline running, and simply fail if it does not pass.

From Command-line Pipeline Linter:

  Jenkins can validate, or "lint", a Declarative Pipeline from the
  command line before actually running it. This can be done using a
  Jenkins CLI command or by making an HTTP POST request with appropriate
  parameters. We recommended using the SSH interface to run the
  linter. See the Jenkins CLI documentation for details on how to
  properly configure Jenkins for secure command-line access.

  Linting via the CLI with SSH

# ssh (Jenkins CLI)
# JENKINS_SSHD_PORT=[sshd port on master]
# JENKINS_HOSTNAME=[Jenkins master hostname]
ssh -p $JENKINS_SSHD_PORT $JENKINS_HOSTNAME declarative-linter < Jenkinsfile

  Linting via HTTP POST using curl

# curl (REST API)
# Assuming "anonymous read access" has been enabled on your Jenkins instance.
# JENKINS_URL=[root URL of Jenkins master]
# JENKINS_CRUMB is needed if your Jenkins master has CRSF protection enabled as it should
JENKINS_CRUMB=`curl "$JENKINS_URL/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,\":\",//crumb)"`
curl -X POST -H $JENKINS_CRUMB -F "jenkinsfile=<Jenkinsfile" $JENKINS_URL/pipeline-model-converter/validate

  Examples

  Below are two examples of the Pipeline Linter in action. This first
  example shows the output of the linter when it is passed an invalid
  Jenkinsfile, one that is missing part of the agent declaration.

  Jenkinsfile

pipeline {
  agent
  stages {
    stage ('Initialize') {
      steps {
        echo 'Placeholder.'
      }
    }
  }
}

  Linter output for invalid Jenkinsfile

# pass a Jenkinsfile that does not contain an "agent" section
ssh -p 8675 localhost declarative-linter < ./Jenkinsfile
Errors encountered validating Jenkinsfile:
WorkflowScript: 2: Not a valid section definition: "agent". Some extra configuration is required. @ line 2, column 3.
     agent
     ^

WorkflowScript: 1: Missing required section "agent" @ line 1, column 1.
   pipeline &#125;
   ^

  In this second example, the Jenkinsfile has been updated to include
  the missing any on agent. The linter now reports that the Pipeline
  is valid.

  Jenkinsfile

pipeline {
  agent any
  stages {
    stage ('Initialize') {
      steps {
        echo 'Placeholder.'
      }
    }
  }
}

  Linter output for valid Jenkinsfile

ssh -p 8675 localhost declarative-linter < ./Jenkinsfile
Jenkinsfile successfully validated.


Understanding DevOps opponents on stakeholder level beyond typical examples What are typical stakeholders and their reasons not to support/enable DevOps principles?*

Background: in small to medium organizations, certain level of DevOps is not a very big problem due to thinner silos and faster communications paths.

But how do you do DevOps at large organizational scales and which groups of stakeholders do you deal with?

I can think of these typical examples:

Developers, either junior or more experienced who would like to "just do their job they are paid for - establishing software solutions and coding them"
Management/Deciders concentrating on the service aspect in terms of man-months, where you find out a thing like a toolchain is non-existent to them: they recognize people doing their work on computers, not more.

Others?

(And how can DevOps be a benefit to them? - as a side thought to be separate question, "how can DevOps be a benefit to stakeholder group X?")

People resist change, particularly when it's perceived to be outside of their control.  Organizations that have been around a while naturally devolve into silos, and organizational structures often reflect that (e.g., database administration teams often manage all of the databases for the org, even though they are often distanced from the business purpose of those databases).  DevOps has the potential to disrupt silos, and may cause realignment of org structures.  For example, you could have product teams composed of developers and operations people (like DBA's) that focus solely on a single product.

For some mid-level managers, that could threaten their sense of job security.  The benefit of a DevOps mindset is that it encourages everyone to understand the value stream to the customer, and the relationship of all the roles in delivering that value.  IT could mean that the manager of DBA's suddenly has to understand development processes because they're now managing a product team.


GKE - GPU nvidia - cuda drivers dont work I have setup a kubernetes node with a Nvidia Tesla k80 and followed this tutorial to try to run a pytorch docker image with Nvidia drivers and CUDA drivers working.

I have managed to install the nvidia daemonsets and I can now see the following pods:

nvidia-driver-installer-gmvgt
nvidia-gpu-device-plugin-lmj84

The problem is that even while using the recommended image nvidia/cuda:10.0-runtime-ubuntu18.04 I still can't find the Nvidia drivers inside my pod:

root@pod-name-5f6f776c77-87qgq:/app# ls /usr/local/
bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src

But the tutorial mentions:

  CUDA libraries and debug utilities are made available inside the container at /usr/local/nvidia/lib64 and /usr/local/nvidia/bin, respectively.

I have also tried to test if CUDA was working through torch.cuda.is_available() but I get False as a return value.

Many thanks in advance for your help.

Ok so i finally made it work.

It is mandatory set a ressource limit to access the nvidia driver, which is weird considering either way my pod was on the right node with the nvidia drivers installed..


How to create a database backup using docker-compose According to this documentation one could create a database backup

docker-compose.admin.yml

version: '3'
services:
  dbadmin:
    build: db_data/
    links:
      - db

running

docker-compose -f docker-compose.yml -f docker-compose.admin.yml \
run dbadmin db-backup

results initially in:

ERROR: Error processing tar file(archive/tar: invalid tar header):

and when it is run for the second time:

ERROR: Error processing tar file(exit status 1): unexpected EOF

The number of downvotes on the link that is mentioned in the question could indicate that there are issues with the instructions.

Perhaps one could use native commands to backup the database. For example, this answer shows how to backup a postgres database that runs in docker using the pg_dump command.


Updating software in a docker container I'm new to this community, so feel free to tell me if this is a dup or belongs in another SE.

I am new to docker, and it's a really cool concept, but I'm struggling with the idea of packaging an app that will be updated later. Suppose, for example, that I've packaged an app that uses a postgres database that I included in the same container. When I build a new image going from postgres 11 to postgres 12, I need to migrate the data directory. How is that normally handled with docker?

My example might be bad, because proper design would have the DB in another container, but suppose I want to upgrade that postgres image, does the new postgres image handle the migration automatically? Is there a common way to run a script when the image is upgraded?

Migrating data directory between major db version can be problematic.

I can suggest to create and restore db dump.
You can upgrade postgresql db container as follow:

Backup database
Shutdown postgres 11 container
Run new postgres 12 container
Restore backup

or single line command:

docker exec postgres-11-container pg_dumpall -U postgres | docker exec -i postgres-12-container psql -U postgres


Deploy helm chart from Jenkins to Kubernetes I want to deploy helm chart from Jenkins to Kubernetes but I don't find any Plugins in Jenkins for Helm.

How can I tell Jenkins to run Helm chart in Kubernetes ?

Use a docker image within your pipeline that has helm and other tools installed.

To quote the docs:

  Pipeline is designed to easily use Docker images as the execution
  environment for a single Stage or the entire Pipeline. Meaning that a
  user can define the tools required for their Pipeline, without having
  to manually configure agents. Practically any tool which can be
  packaged in a Docker container. can be used with ease by making only
  minor edits to a Jenkinsfile.

It then gives an example that we can modify to show running helm:

pipeline {
    agent {
        docker { image 'your_image:1.0' }
    }
    stages {
        stage('Deploy') {
            steps {
                sh 'helm list'
            }
        }
    }
}

This is a total game changer. Docker is the plugin to end all plugins. Before Docker you had to find a plug-in written in Java that to exposes a DSL to be scriptable in a Groovy sandbox. That was because it was not secure or scalable to let you simply run your own tools and scripts on a shared Jenkins. You were not even allowed use the full power of Groovy or Java. With Docker it is safe run any script or tools. So you no longer have the indirection of Java plugins and you can use the full power of any Linux tool on any Linux distro.

This approach is exactly how new cloud based build engines work. With bitbucket.com pipelines and cricleci.com you don’t need any plugins. You simply say which containers to use and what commands to run within them just like the example above. The build engine simply checks out the code where the docker container can see it.

Once you have used Docker based builds for a while doing it the old way feels very restricted and limiting. For example consider having to move files around in the build. In a pipeline with Groovy if you are new to Jenkins you have to look at a huge amount of (very ugly) documentation to just to basic things. If you use docker you can simply use standard bash and Linux commands.


How does Docker Swarm balance tasks? From docs.docker.com - force-the-swarm-to-rebalance I can see that Docker Swarm "tasks are given to less busy nodes" to reach eventual balance.
I'm trying to find more info on what "busy" means exactly. For us, running a lot of Java microservices, our primary resource is memory. We are constantly running into OOM killers on the hosts as Swarm keeps scheduling containers to nodes that are already very low on RAM.
I have a 7 nodes swarm in front of me right now with one node's top stating:
top - 22:55:04 up 234 days,  8:48,  5 users,  load average: 0.53, 0.68, 1.09
...
KiB Mem : 18330008 total,   425268 free, 13718032 used,  4186708 buff/cache
KiB Swap:   511996 total,   389836 free,   122160 used.  4225292 avail Mem

This is the lowest in free/available memory in the entire swarm. Yet this node ALWAYS gets the task I'm scheduling. Other nodes are much much more relaxed:
top - 22:56:28 up  5:49,  2 users,  load average: 0.38, 0.61, 0.79
...
KiB Mem : 18329940 total,  4899940 free,  7877052 used,  5552948 buff/cache
KiB Swap:   511996 total,   511996 free,        0 used. 10098352 avail Mem

Yet they will never get the task. None of the other nodes will. Only the above mentioned node will, which has by far the lowest available memory.
So what criteria does Swarm use to determine the "business" of a node to determine if a task should be scheduled on it? Available memory clearly does not seem to be a metric in the decision.
Sidenote: I am aware that I can assign resource limits to my containers to prevent the node from being over-scheduled. This is what I am doing right now and I assume it will fix my problem. But I want to know about Swarm's inner workings to determine where to schedule and where not to.

AFAIK, Swarm does zero real-time resource monitoring. It's only as smart as reservations and limits that you set on the Service beforehand.
First, it takes into account if the tasks of that Service are spread out. It will try not to schedule multiple tasks on the same node, and may ignore total task count per node to do so.
Then, it counts the number of tasks per node and finds the one with the least task count.
All these is restricted by resource reservations. If that node doesn't have the room (based only on the reservations you set), it'll look to the next lower task count.
This is also why Swarm alone can't do built-in auto-scaling because it has no way to track metrics over time.
It does know total CPU and memory, so it knows how to track "full" once you've set resource reservations.
More info: https://github.com/docker/swarmkit/blob/master/design/scheduler.md


What is docker? what problems does it solve? I'm a jr. developer and I'm being required to learn docker. I googled around and read a bunch of articles about docker but couldn't really understand what it was for, what problem does it solve.

Example about GIT. When I was at university when we had our thesis, my groupmates and I use facebook messenger or email to pass parts of the code, we indicate what file is it for and on what line, so I understand how GIT resolves a problem. but what about docker? When I deployed to AWS, I ssh thru the server, git clone your repo, do an npm install, so whats the difference? I still have to do that right but with one command? docker compose run or something? so thats it? hope someone can explain it to me. thank you and have a good day.

Docker is a technology that allows you to create OS-level virtualization for software delivery.  There are many, many articles written about what this technology is, how it can help developers and companies, and tutorials on how to get started.

To answer your question of why as a developer you may care (non-exhaustive list):

Building an application that is immediately ready for deployment.
Removing cumbersome installs for quick prototyping and testing.  I use it all the time to setup a SQL server to test scripts before deploying them.
Removes the "It works on my machine" since all the application dependencies are in a "container".
Allows you to easily run and port services that usually would require several minutes of installations and configurations.

For your AWS example, imagine not having to SSH into a server, pull source down, and setup the environment every time you want to scale up or out.  Most cloud providers now have options to deploy your containers and not worry about all those steps (see AWS ECS).

The Docker-Compose you mentioned is slightly different in that it is a way to easily run multiple containerized services together.


How does 'Environment Management' work in a DevOps world? I've had some very interesting conversations today about the role of Environment Management within teams that follow DevOps practices.  The traditional role of an Environment Manager was someone who:

  Took overall responsibility for coordinating multiple teams and components at all stages of the software development life cycle, SDLC, to ensure that a known sets of software versions are delivered and available in a timely manner.

It is my belief that in a cross-functional Agile team following DevOps practices these responsibilities belong to the team as a whole, ultimately being accountable to the Product Owner as the representative of the business.

In short, my question is who is... responsible for Environment Management in a team following DevOps practices, i.e., one that does not have an external Environment Manager function available to them.

I've never heard of an "Environment Manager."  Release Management, on the other hand, has historically been consolidated to a single person or team.

In a DevOps model, Release Management is more of a process supported by elements from both Dev and Ops.  Organizations at high levels of "DevOps maturity" don't really have a distinct Release Management event- the act of releasing is baked into the Continuous Delivery pipeline (see "push on green" What is "Push on Green"?)

It's possible the OP is also referring to Configuration Management, where environment-specific configurations are managed manually, or by tools like Puppet or Chef.  That work is traditionally an Ops-managed task, with higher-maturity orgs handling that through code.  Responsibility may be shared between Dev and Ops, or there may be a dedicated SRE team that deals with Config Management.


Ansible: Run tasks on specific hosts with var/fact from another host I'm trying to get the output from a command on the first host in group managers and execute it on the other hosts in the same group.
hosts file
[managers]
k8s01
k8s02
k8s03

[workers]
worker1
worker2
...

main.yaml
- name: Create token to join cluster
  shell: kubeadm token create --print-join-command
  run_once: true
  changed_when: false
  when: inventory_hostname == groups['managers'][0]
  register: kubernetes_join_command_result

-name: Set join command for managers
- set_fact:
    kubernetes_join_command_controlplane: "{{ kubernetes_join_command_result.stdout }} --control-plane"
  when: kubernetes_join_command_result.stdout is defined
  delegate_to: "{{ item }}"
  delegate_facts: true
  with_items: "{{ groups['managers'] }}"

- name: Debug var
  debug:
    msg: 'Join Command: {{ kubernetes_join_command_result.stdout }}'

- name: Join managers into cluster
  shell: "{{ kubernetes_join_command_controlplane.stdout }}"
  args:
    warn: no
  when: inventory_hostname != groups['managers'][0] and inventory_hostname in groups['managers']

Instead of running the join command in hosts k8s02 and k8s03, I get the output below.
TASK [managers: Join managers into cluster] ********************
skipping: [k8s01] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [manager: another task] ***************************

The condition works as expected. For example, the inventory and the playbook below
shell> cat hosts
[managers]
k8s01
k8s02
k8s03

[workers]
worker1
worker2

shell> cat playbook.yml
- hosts: all
  gather_facts: false
  tasks:
    - debug:
        var: inventory_hostname
      when:
        - inventory_hostname != groups.managers.0
        - inventory_hostname in groups.managers

give
shell> ansible-playbook -i hosts playbook.yml

PLAY [all] ****

TASK [debug] ****
skipping: [worker1]
ok: [k8s02] => {
    "inventory_hostname": "k8s02"
}
skipping: [k8s01]
skipping: [worker2]
ok: [k8s03] => {
    "inventory_hostname": "k8s03"
}


JFrog on dedicated disk Is it possible to install JFrog on a separate disk? For example in an AWS EBS

I find their documentation a bit confusing, so i checked don't have a walk-through for this.
I'm looking at this documentation: Installing on Linux Solaris or Mac OS.

It is.

You can install the ZIP installation and unzip it in the EFS.
Alternatively, install using Debian/RPM and mount the EFS to /var/opt/jfrog/artifactory in advance.

Alternative approach can be to only configure the Filestore to use the EFS.


How do I trigger scripts on a merge or when a branch is deleted? I  am using jenkins multibranch pipelines. If a branch is merged and deleted jenkins removes the branch from the pipeline as well.

This is great, but I want to run a command/script when a branch is removed, how can I have jenkins run a command when a branch is removed? Is there a way to have jenkins trigger a job before it removes the branch?

Edit: I should add I'm using bitbucket. It would be neat if Jenkins could trigger an event when purging a branch of a multibranch pipeline. Ideally it wouldn't matter what my git provider is.

"This is great, but I want to run a command/script when a branch is removed, how can I have jenkins run a command when a branch is removed?"

  You will need to use a webhook for this, I use the following
  settings in Github:

  Bitbucket only supports the following webhooks:

"Is there a way to have Jenkins trigger a job before it removes the branch?"

  Not that I can see, but in Github you can create a custom label and
  trigger a job via a webhook when the label is added to a branch.


Ansible resetting swap linux I am newbie to Ansible, i failed using ansible via command that way :

ansible host -a "/sbin/swapoff -a && /sbin/swapon -a"

Only this worked

ansible host -a "/sbin/swapoff -a
ansible host -a "/sbin/swapon -a

I am thinking of making a playbook, starting maybe with this:

- name: Turn off swap
  shell : "swapoff -a"

- name: Turn on swap
  shell : "swapon -a"

I would like to reset swap space with a condition, only when the swap used space is greater than 100mb. Is it possible to do that ?

Many thanks for your help

You can certainly do this through a handler or task, but first you need to generate the trigger conditions. The swap space is in the Ansible facts that are returned when setup: is run. From the Ansible docs:

  Facts

  These are variables that contain information pertinent to the current host (inventory_hostname). They are only available if gathered first.

  ansible_facts
     Contains any facts gathered or cached for the inventory_hostname Facts are normally gathered by the setup module automatically in a play, but any module can return facts.

  ansible_local
     Contains any ‘local facts’ gathered or cached for the inventory_hostname. The keys available depend on the custom facts created. See the setup module for more details.

As an example:

ansible -i localhost, all -c local -m setup -a filter=*swap*
localhost | SUCCESS => {
    "ansible_facts": {
        "ansible_swapfree_mb": 437,
        "ansible_swaptotal_mb": 979
    },
    "changed": false
}

In your playbook your task to reset swap should then have the conditional like:

when: ansible_swap_free_mb < 500

In your case, however you want to trigger on swap used, so you would have to compute the difference between total and free.