Is there a way to set build parameters outside of the pipeline directive? My build parameters are defined in my jenkinsfiles

All builds run on container slaves on google kubernetes engine cluster

This means users need to wait for a container to start before anything happens, including just updating the "Build with parameters" choices.

Can I set parameters outside of the pipeline directive so it doesnt need to start a container to update the parameters?

or even better maybe there is a way to have Jenkins run functions in my jenkinsfile to get the updated parameters when it scans branches?

EDIT:
It might have a chance for me to misread your question at the first time, so let me add another one.

If you meant by Jenkinsfile as you have a manual parameter input in Jenkinsfile for Multibranch or Github Org pipeline, then probably bad idea.

Those kind of jobs should run with predefined parameters or take them on-the-fly like manipulating some environment variables without user input.

You can simply make another pipeline job for manual trigger if the predefined parameters can't cover all the case you need.

And for run function, it looks like build job is what you are looking for.

https://jenkins.io/doc/pipeline/steps/pipeline-build-step/#-build-%20build%20a%20job

build job: 'Another job in same Jenkins server', parameters: [string(name: 'param1', value: ${new_value})]

--------------------------------------

Are you sure the parameter input is waiting for slave containers?
or do you use some special choice plugins for build parameter?

I think a job is even not starting until a user fills out the parameters and hit the build button in Jenkins UI - so no container required until then.
We had same k8s slaves, but never had that issue.

(But some parameter plugins might need some pre process..)


How to get other than "" when interpolating {{ .Environment.SOME_VAR }} (with SOME_VAR set) into cache key? Using this configuration:

version: 2
jobs:
  test:
    docker:
    - image: nixorg/nix:circleci
    environment:
      NIXPKGS_REV: 3c83ad6ac13b67101cc3e2e07781963a010c1624
    steps:
    - restore_cache:
        name: Restore Nix Store Paths
        keys:
        - paymentserver-nix-store-v1-{{ .Environment.NIXPKGS_REV }}
        - paymentserver-nix-store-v1-

The job tries to restore the cache using the key paymentserver-nix-store-v1-<no value>.

https://circleci.com/docs/2.0/env-vars/#setting-an-environment-variable-in-a-job says this is the correct way to set an environment variable.  https://circleci.com/docs/2.0/caching/#using-keys-and-templates says interpolating environment variables this way is supported for cache keys.

What's going on?  How do I use the value of an environment variable in a cache key?

A closer reading of the documentation about using environment variables in cache keys reveals the issue (emphasis mine):

  The environment variable variableName (supports any environment variable exported by CircleCI or added to a specific Context—not any arbitrary environment variable).

"exported by CircleCI" seems to mean the list of mostly-CIRCLE_...-prefixed variables (documented as "Built-In Environment Variables") while (unsupported) "any arbitrary environment variable" includes variables set in an environment section in config.yml.

I ended up working around this limitation something like this:

jobs:
  build:
    docker:
      - image: "nixorg/nix:circleci"

    environment:
      NIXPKGS_REV: "3c83ad6ac13b67101cc3e2e07781963a010c1624"

    steps:
      - "run":
          name: "Prepare For Cache Key"
          command: |
            echo "${NIXPKGS_REV}" > nixpkgs.rev

      - restore_cache:
          name: "Restore Nix Store Paths"
          keys:
            - paymentserver-nix-store-v1-{{ checksum "nixpkgs.rev" }}
            - paymentserver-nix-store-v1-

That is, instead of trying to use the value of NIXPKGS_REV from the environment in the cache key, an earlier step writes the value of that environment variable to a file and then the cache key is constructed using the checksum of that file.  Computing file checksums for use in cache keys is well supported so this produces a cache key which is derived from the value of the environment variable.  The key doesn't contain the value of the environment variable but since the key's only purpose is to be unique for certain unique inputs, using the checksum works just as well.


Secure way of uploading data to Google Cloud Storage We're making some machines in which there's a part which uploads the images captured by the camera to Google Cloud Storage. For this purpose what I've done is

Create a service account for each machine.
Create a custom role with permissions:

storage.objects.create
storage.buckets.get
storage.objects.get

Apply this role to that service account.
Download the JSON credentials key file and use this file with python script (in which I specify bucket name) to upload image to GCP Storage.

Is this way of doing things efficient and secure given that we only ship 2-3 machines each month?
Also I will have to ship JSON file with each machine, if the above method is valid, is this fine or there's any method to hide this key file?

Yes, your approach sounds okay to me. I am not sure how you "ship" machines but it is a good idea to use a service account per machine as you can delete the service account key if the machine ever becomes compromised.
You can, and should, fully automate the process of generating the credentials key files using the Google APIs or gcloud cli.
It is best practice to periodically rotate (re-issue) credentials key files. This is in case they are accidentally leaked in a way that you do not notice. If you always revoke the key and issue a new one every month then it limits the total damage somewhat.
If you do rotate keys then you have a key distribution problem. How do you securely distribute the new credentials key files onto machines you shipped in the past? You can use GnuPG to create a public-private key pair on every host. Each month you can run a script that generates a new credentials key file per machine, encrypt it with the machine's public GPG key, then delete the old service account keys.
The script that pushes images to a bucket must then periodically "phone home" to collect a fresh GPG encrypted credentials key file. It can then decrypt it with its own private key. One way to do this is to put the encrypted keys into a different bucket. Create a single service account that can only read that bucket. Create a single credentials key file for that service account that you install onto every machine. That key can then only be used to only collect encrypted keys that can only be decrypted on the correct host.


Jenkins Pipeline steps that run before Jenkinsfile I'm just starting to get my toes wet with Jenkins and CI/CD.

I've built a set of static analysis tools and corresponding configurations I want to run before a branch's Jenkinsfile is even considered. I want to have Jenkins run these tools across every branch regardless of what the Jenkinsfile on the branch says. I want to make sure that the static analysis is out of the purview of the repo. How can I build a set of steps that are run before the Jenkinsfile on the branch is considered?

I see (at least) two possible solutions that both have their drawbacks.

Using a shared library containing all the code for your static analysis. That code can then be called as a step in the pipeline.  https://jenkins.io/doc/book/pipeline/shared-libraries/. This doesn't keep the execution out of the pipeline completely, but it keeps the code out of each pipeline/branch and is (in my opinion) a pretty straight forward solution.
Putting the static analysis into a separate job, trigger that job for example with a git hook and then let that job trigger the pipeline. I my opinion a more complex solution that has more moving parts, but may be closer to what you're asking for.


What courses would be useful for a DevOps Engineer? I am a 'DevOps Engineer' based in the UK. I have 2.5/3 years experience in the IT field; starting out as an apprentice now a mid-level engineer learning/practicing Linux/Windows System Administration, Continuous Delivery (Jenkins), Continuous Integration/version control (Git, Visual Studio), AGILE/SCRUM methodology, configuration management (SaltStack), virtual machine technologies (AWS EC2, VMware etc) as well as S3 and BASH/PowerShell scripting.

I haven't got any formal qualifications nor do I think it is necessary to excel in the field. However my manager has asked myself and my colleagues to find some courses and/or qualifications which will help us become fully-fledged DevOps Engineers and I think it'd be stupid not to do so!

Could anyone recommend any useful courses/qualifications?

Linux Academy has several webinars and tutorials (including sandbox VMs for hands-on experience) in several technologies favored by DevOps shops including AWS, Docker, Ansible, Jenkins, Puppet and many more.


how to run Docker/Jenkins behind a proxy For the past few days I’ve been scratching my head trying to get the Jenkins docker image to work

The issue appears to be a proxy issue.

  OS: Ubuntu 16.04 LTS

  Running behind work proxy

  Docker version 19.03.5, build 633a0ea838

  Jenkins:https://hub.docker.com/_/jenkins/

When I run the following command, I can’t access jenkins on localhost:7000

docker run --name myjenkins -p 7000:8080 -p 50000:50000 -v /var/jenkins_home jenkins

However I can with the following command

docker run --name myjenkins -p 7000:8080 -p 50000:50000 --env HTTP_PROXY="http:// localhost:8080" --env HTTPS_PROXY="https:// localhost:8080" -v /var/jenkins_home jenkins

The issue is, with the second command, I can access the Jenkins but I can’t install plugins and it asks me to configure the proxy
Setting the proxy in the UI didn’t work

My understanding is that Jenkins look for port 8080 by default and because in the command above I’m using port 7000, this is preventing Jenkins from connecting to the internet (correct me if I’m wrong)

So what I did was download the Dockerfile for Jenkins and changed the port from 8080 to 7000

Dockerfile

I am now attempting to build the image with the following command

docker build --tag jenkins:1.0 .

I get the following error

curl: (7) Failed to connect to github. com port 443: Connection timed out
The command '/bin/sh -c curl -fsSL https:// github. com/krallin/tini/releases/download/v${TINI_VERSION}/tini-static-amd64 -o /bin/tini && chmod +x /bin/tini && echo "$TINI_SHA /bin/tini" | sha256sum -c -' returned a non-zero code: 7

I suspect, this is a proxy issue and I did put proxy arguments in the Dockerfile but not sure if this is what I need to do or whether I’m doing it correctly

Any idea what could be the solution?

Thank you

I believe that the problem lies on your proxy environment variables.

$ docker run \
    --name myjenkins \
    -p 7000:8080 \
    -p 50000:50000 \
    --env HTTP_PROXY="http:// localhost:8080" \
    --env HTTPS_PROXY="https:// localhost:8080" \
    -v /var/jenkins_home \
    jenkins

The values of $HTTP_PROXY and $HTTP_PROXY are not supposed to have space characters on them and the protocol should be http. Additionally, although setting $HTTP_PROXY is useful for compatibility with other applications running inside the container, curl expects the HTTP proxy environment variable to be written in lowercase, that is, $http_proxy. From curl (1) manual page:

  Environment

  The environment variables can be specified in lower case or upper case. The lower case version has precedence. http_proxy is an exception as it is only available in lower case.

  http_proxy [protocol://]<host>[:port]
  Sets the proxy server to use for HTTP.

  HTTPS_PROXY [protocol://]<host>[:port]
  Sets the proxy server to use for HTTPS.

  FTP_PROXY [protocol://]<host>[:port]
  Sets the proxy server to use for FTP.

  ALL_PROXY [protocol://]<host>[:port]
  Sets the proxy server to use if no protocol-specific proxy is set.

  NO_PROXY <comma-separated list of hosts>
  list of host names that shouldn't go through any proxy. If set to a asterisk '*' only, it matches all hosts.

Therefore the docker run command might be written as below:

$ docker run \
    --name myjenkins \
    -p 7000:8080 \
    -p 50000:50000 \
    --env HTTP_PROXY="http://localhost:8080" \
    --env http_proxy="http://localhost:8080" \
    --env HTTPS_PROXY="http://localhost:8080" \
    --env https_proxy="http://localhost:8080" \
    -v /var/jenkins_home \
    jenkins

However, my solution will certainly not work because the port 8080 inside the container is being used by Jenkins and not by a HTTP proxy. Please, could you provide more information on how your work proxy is set up so I can edit my answer with more accurate details? Are you running a HTTP proxy such as squid or cntlm in your computer that receives requests from applications running in your workstation and forwards them to the corporate proxy after authentication? If so, you will need to adjust container's network configuration according to Stack Overflow question From inside of a Docker container, how do I connect to the localhost of the machine?.

EDIT: According to comments, a cntlm instance running locally on port 8080/tcp provides a HTTP proxy service that local applications must use in order to access internet. I believe that the least complex way to expose it to Jenkins container is by specifying --network=host parameter and sharing the root network namespace with the Jenkins container. However, Jenkins is set to listen the port 8080/tcp by default, therefore it must be instructed to listen to an alternative port, such as 7000/tcp. That can be achieved by passing --httpPort=# command line argument to Jenkins via JENKINS_OPTS environment variable.

EDIT 2: While analyzing the repository code, I've realized that Jenkins is a Java application and I am not sure that JVM recognizes $http_proxy and $https_proxy environment variables properly. So, to ensure that Jenkins will use cntlm as HTTP proxy, I also suggest passing proxy configuration properties to the JVM via the JAVA_OPTS environment variable.

EDIT 3: The Docker image jenkins is deprecated in favor of the jenkins/jenkins:lts one. The former image is currently unmaintained and seems to contain a version that is outdated and incompatible with plugins being downloaded.

To sum up, the docker run command have to be written as below for Jenkins to work in your environment:

$ docker run \
    --name myjenkins \
    --network=host \
    --env HTTP_PROXY="http://localhost:8080" \
    --env http_proxy="http://localhost:8080" \
    --env HTTPS_PROXY="http://localhost:8080" \
    --env https_proxy="http://localhost:8080" \
    --env JAVA_OPTS="-Dhttp.proxyHost=localhost -Dhttp.proxyPort=8080 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8080" \
    --env JENKINS_OPTS="--httpPort=7000" \
    -v /var/jenkins_home \
    jenkins/jenkins:lts

Passing --env JAVA_OPTS="-Djava.net.useSystemProxies=true" instead might work. I am not able to perform a try, though.


What are the main differences between Jenkins and TeamCity if one is used to working with Jenkins? These tools seem to share very similar characteristics.

How complicated would it be to start using TeamCity after getting used to working on Jenkins? Are there specific concepts one needs to be aware of?

TeamCity :

It does look nicer, if this is important for your team then it should definitively weight in.  That said, if it is VERY important than you probably will end up creating tools or some sort of dashboard overlay to support your team at which point what you really want is the one with the best API.  Have not tried Jenkins API so I cannot compare, the TC API though should get you what you need.

Their support is pretty good, the guys respond relatively fast and are courteous.  However that does not mean you will get what you want.  If you use the system in an unconventional way you can very well get your bug put on the shelves... happened to us.  At which point it becomes quite frustrating to use, you are faced with a black box leaving little alternative than to work around it.  At this point things can get hackish and ugly.

It is generally pretty quick to get done what you want and the log interaction quasi API is a very nice feature if you do a lot of custom scripting in your pipeline.

Jenkins:

Battle tested and Extensive.

But it is somewhat less pretty, would not go as far as saying it is ugly though, one could say that functionality comes before appearance.

I'm pretty sure you can find private paid support plan by third party companies if you look around.  If this is important for your shop don't just block at the "OpenSource" part of the deal, the community is quite expansive.

Lots, I mean LOTS of plugins.  Again don't just just limit yourself to the official channels, lot more plugins can be found in github and other places.

I found both to be about equally fast to get started with, though with Jenkins you might need to be more dynamic with plugins than with TeamCity.  So if you have a tight IT department and wont get admin access to the server this could pose a problem.  Compounded by their much faster release cycle than TeamCity (weekly).

I found that Jenkins does support more release cycle paradigms than TeamCity.  Might be easier to find a process template out of the box that is a closer match to what you have in mind. I say this with reserves as I have not dealt with TeamCity for 2 years now.

I personally prefer Jenkins mostly because I am biased towards Open Source for such tools and in a lesser degree because I found it's structure and config mechanism agree more with me.

YMMV


When to put all containers in a pod and when to deploy them individually? One of the reasons for asking this question is that it seems to be impossible to access a container using kubectl exec when these are deployed in a pod.

Let's start from the beginning.

First, I'll answer the original question, the one in the title.

When to put all containers in a pod and when to deploy them individually?

This depends on what your app does. For instance (this is an actual example from the company I work for), for a web application, you may have a pod which runs three containers:

app: the actual app with its code
nginx: serves all the requests
heka: a log service which proxies the logs to wherever you want them

The reason for grouping this three containers within a single pod is simple: a pod assures you the containers run in the same Kubernetes node, this is to have the least possible latency with requests being interchanged along the three containers.

That being said, this is only an example, you could have different and multiple reasons for grouping containers in a pod, but I think this is the most common.

With that being clarified, I follow to the question that you present in the body of your post.

How do you access a single container within a pod?

This is simply using kubectl, I use the command like this:

kubectl exec -it <pod-name> -c=<container-name> -- /bin/bash

Suppose we want to access the nginx container inside our my-app pod; we would type this:

kubectl exec -it my-app -c=nginx -- /bin/bash

If you don't specify the container with the -c flag, it defaults to the first container that is in the Kubernetes yaml (I think). So when you say it was impossible to access a container, you actually did access a container (the default one), because a pod cannot be accessed, it is merely a collection of containers and its configurations.

Hope I've been helpful.


Is cross-region replication 100% fool-proof for S3 region outages? Amazon S3 has an option of cross-region replication which should be pretty fault-tolerant against region/zone outages.

Does that mean those who are ranting about the outage did not make use of this aspect?

Or is that cross-region replication is not completely fool-proof and would not have helped?

The drawback when there's replication come from the note below:

  Amazon S3 routes any virtual hosted–style requests to the US East
  (N. Virginia) region by default if you use the US East (N. Virginia)
  endpoint (s3.amazonaws.com), instead of the region-specific endpoint
  (for example, s3-eu-west-1.amazonaws.com).

When you use replication you usualy let AWS takes care of routing the alias to one region by targetting s3.amazonaws.com in your REST request from your servers and let the redirect do it's job.

Whenever N.Virginia is down, the magic cease to work and you're out of luck to access your data and have to update your configuration to choose a specific region endpoint.

The problem does not come from the DNS (a request to the bucket itself will work) but from S3 clients, which will connect to the S3 API endpoint before accessing the bucket, in this case the dns resolution is done on s3.amazonaws.com and this is us-east-1 endpoint.

When you use regions alias, you loose the ease of load balancing over regions with the health check from AWS included.

If you use DNS cname targeting the regions to switch quickly, you're responsible of your DNS TTL but nothing guarantee cache servers of client ISP will honor your value (one of many cache your client may encounter).

And lastly, if you try to load balance yourself you'll probably create the same SPOF than AWS already have with the added burden of maintaining it.

AWS is working on it but that's all the information I have at time of writing.


Additional push event syncing between Artifactory instances We have two instances of Artifactory for Prod and Non-Prod.

Normal syncing happens via a pull mechanism by the Prod instance.

Is it possible to also have an additional push event sync from Non Prod instance to the Prod that we would use for hotfixes.

Yes, you can set up push event-driven replication. Please note that while pull replication prepopulates remote repository in server A from a local repository in server B, push replication replicates between two local repositories. Once you set up a virtual repository it doesn't really matter because you'll have a single point of access anyway, but just something for you to be aware of.

You can read more about push replication and pull replication in the user guide. Also, please find here a whitepaper on different replication topologies and a webinar on the same topic.


Pass environment variables from pipeline job A to pipeline job B I passed params to pipeline B and set the environment variables from parameters. However, "${params.URL}" is evaluated as null.

Pipeline A is declared in Jenkinsfile:

pipeline {
    agent any

    stages {
        stage('Run on all repos') {

            steps {
                script {
                    sh 'ls'
                    build job: 'run-on-single-repo'
                    parameters:
                            [string(name:'URL', value:'val')]
}}}}}

Pipeline B is declared in run-on-single-repo.groovy:

pipeline {
    agent any

    stages {

        stage('Find missing dependencies') {

            environment {

                URL = "${params.URL}"

            }

            steps {
...

Both pipelines are configured in Jenkins as 'Pipeline script from SCM' and work.

How to pass environment variables from pipeline job A to pipeline job B?

The issue was missing comma between build job and parameters. Below is the correct syntax:

...
                    build job: 'run-on-single-repo',
                    parameters:

...


From a security perspective, to which location should one copy an app inside a docker image? I use docker to deploy a lot of apps since a couple of years. Today I ask myself, should I copy my app to a different location in the docker image? I used to copy the file to the root, e.g. COPY . /AppName, but I wonder whether this is secure.

In order to find an answer to this question, this tutorial was read, but it does not clarify where to copy a app inside a docker image and why.

I don't see a point.

An external attacker (who has access to the host) can easily find where the files are - he only has to look at the last file system layer and will have them presented right there for him.

An internal attacker (who gains access to the container through a misconfigured part of the app or whatever) will also rather easily find them by looking through the process table at /proc and noting the CWDs.

Attacking the image might be easier with a fixed location, as the attacker only has to add another FS layer, overwriting that directory with his own application. But then, if you did not secure your image store, it does not matter anyways what else you do.

So... I'd say the convenience of a common location is good, and little to no security would be gained; and if, then only security by obscurity, which does not count.


Azure Release Pipeline Fails Using Expand-Archive Command Sometime in the last week, our pipeline started using Expand-Archive to open the .zip file during the "IIS Web App Deployment" step instead of 7-Zip. It does not appear that this was done by our team, so I'm presuming this may have been changed in DevOps.
We are deploying to Server 2012R2.
Several of our apps are okay with this, however one of them fails to expand. It seems as if this might be related to path length (this app uses some very deep folder nesting with Angular) since I can manually run the command successfully if I shorten the destination path.
The command that fails to runs on the server is:
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -NoProfile -NonInteractive -Command "Expand-Archive -Path "C:\azagent\A3\_work\r1\a\thisistheazurejobname\drop\thisisourappsnamex.zip" -DestinationPath "C:\azagent\A3\_work\_temp\temp_web_package_8157628602055582" -Force"

If I reduce it to:
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -NoProfile -NonInteractive -Command "Expand-Archive -Path "C:\azagent\A3\_work\r1\a\thisistheazurejobname\drop\thisisourappsnamex.zip" -DestinationPath "C:\azagent\A3\_work\_temp\short" -Force"

...it appears to work fine.
Obviously, I am speculating that it's a path-length issue, but evidence does seem to support it. Also, I reconstructed what would have been the 7-Zip command and it worked as expected:
C:\azagent\A3\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.184.0\node_modules\webdeployment-common-v2\7zip\7z.exe x -oC:\azagent\A3\_work\_temp\temp_web_package_8157628602055582 C:\azagent\A3\_work\r1\a\thisistheazurejobname\drop\thisisourappsnamex.zip

I looked at the advice here and found we are already using the setting:
https://igorpuhalo.wordpress.com/2019/08/29/overcoming-long-path-problem-in-powershell/
My hope is that someone knows what may have changed, can suggest a straightforward way to go back to using 7-Zip, or a way to make Expand-Archive to work.

This was answered via StackOverflow by Kevin Lu:
https://stackoverflow.com/a/66739852/1224614
In short, the second suggestion he made to update to PowerShell 7 and adding the environment information worked. I also had to run the script to reregister the Azure agent on the target server.


Check a process is running with Salt Stack? Is it possible to check if a specific process is running via a Salt State? Looking through Salt Stack documentation and scouring forums I haven't found any way of simply checking if a service is running and reporting the output.

I have however found this:

httpd-absent:
  process.absent:
    - name: apache2

The opposite of this would be ideal for my needs e.g:

httpd-present:
  process.present:
    - name: apache2

Is anyone aware of a method in Salt which would check if a defined process is running based on a name or pattern (regex) and simply report back with a true/false or similar?

It's a bit hacky, but you can always do something like:

check_process:
  cmd.run:
    name: ps aux | grep '[f]oobar'

The exit code will be non-0 and the state will fail if foobar doesn't exist.


Chef Provisioning for canary deployment and orchestration I am looking for best tool that, does below jobs for deployment.

Configuration:

1) Configuration of deployment node machines in specific environment

2) Configure no of service instances to be alive in environment at all time

Deployment:

Now, Just doing above configuration. When I trigger deployment of N services. It will randomly pick up nodes from deployment environments and will start total N services.

Multiple Services:

If I have 2 nodes and I want to bring up 4 services, it should bring up 2 services on each node.

Service Failure recovery:

If any machine goes down or any service goes down in any node. It will bring up a new service in any of environment node.

Scenario to handle:

Choosing nodes at random used in distributed system with system discovery/failure.

1) Service deployment :

I required 2 services of calculator, and other services as well. And I have total 4 machines. Now 2 calculator services will be deployed in any node.

2) Service discovery/failure:

These 2 calculator services will be registered with service registry. Now my application will query to service registry, and service registry will provide node details with service running. Same in failure case, service will be started in any node, and discovered by application through service registry.

You should take a look at BOSH.  Its the tool that is used by CloudFoundry, its services and a distro of Kubernetes called Kubo for installation,  management and "Day 2" operations.

It's basically a declarative, cloud-agnostic orchestration tool that features rolling updates, canary deployments, scaling, monitoring and self healing.  It can monitor VMs as well as processes on those VMs (i.e. services) and will make sure that the deployment is running as you specified it to in the deployment manifest.

In order to do all of this (especially the monitoring and self-healing bits) it has a client-server architecture which is deployed with a cut-down version of BOSH itself called bosh bootloader or bbl for short.  You use this to deploy the BOSH director.  You talk to the bosh director by installing the bosh-cli (brew install bosh-cli on a mac).

For you deployment you would first need to create what's called a BOSH release and this can, admittedly, be a little daunting if you are not familiar with BOSH but as CF, its services and Kubo are all open source there are tons of references out there.  There are also lots of pre-backed releases and stemcells (OSes).

After creating your release you upload it to your bosh director and bosh deploy.  To upgrade it you upload the next version of your release and bosh deploy.  To patch a security vulnerability you upload the latest blessed stemcell from bosh.io and bosh deploy.  I am sure you get the picture.

If you want to go the next level then there is a good getting started guide here.


gitlab kubernernetes integration: configuring the installed runner I connected my existing kubernetes cluster to gitlab using the gitlab integration in myproject/operations/kubernetes on the left tab pane in the gitlab gui.
After having installed helm + gitlab runner from the gitlab GUI applications, the new gitlab runner is visible in the project runners.
What I don't understand from the gitlab documentation is how do you modify  the new runner's configuration. You cannot use the gitlab-runner cli, like for a non-kubernetes runner, because the runner in running in a pod, not on the host machine, right?
I am not sure which part of the runner doc applies to my case. What to use if I wanted to make the runner use another executor or build from another image ? (default image of the pod is ubuntu 16 for me).
If you have already installed and configured a runner installed this way, from the GUI, please tell me how. Thank you.
Gitlab 12.9.2 hosted

Installing kubernetes integration on gitlab
In Admin Area -> Overview -> Runners, the instructions they give are:

Install GitLab Runner
Specify the following URL during the Runner setup: https://<your gitlab hostname>/
Use the following registration token during setup: <insert-token-here>
Start the Runner!

I'm on version 13.13.0 but I don't remember the instructions changing from version 12.x.x.
Project specific runners
As for your question regarding configuration, I'm a little bit confused if you're looking for something beyond those instructions. If you want to set up a runner specific to a project, you can go to a project then navigate to Settings -> CI/CD -> Runners -> Specific Runners. There, there are more instructions on setting up project-specific runners.
Further configuring the runner
EDIT: The executor entirely depends on how the runner itself is installed. If it's installed on a kubernetes cluster, then it must be the kubernetes executor. That will affect your pipeline. I think you can create another one if you install another one with the runner chart. You can also use Helm Tiller.


Mount Azure FileShare using terraform What is a good way to mount an Azure File Storage automatically during provisioning on VMs?

There is a Microsoft doc describing how a file share can be mounted in general. This requires that Azure CLI is installed and a user is logged in.

As an alternative, I have created a working setup using terraform working on local host:

First, the existing storage account needs to be defined:

data "azurerm_storage_account" "storage-account" {
  name                = "storageaccount"
  resource_group_name = "RG-storage-account"
}

When a VM is provisioned, the share is mounted by a remote execution provisionier:

resource "azurerm_virtual_machine" "vm" {
  # ...
  # stuff

  provisioner "remote-exec" {
    inline = [
      "sudo mkdir -p /mnt/${data.azurerm_storage_account.storage-accoun.name}/share",
      "sudo mount -t cifs //${data.azurerm_storage_account.storage-account.name}.file.core.windows.net/share /mnt/${data.azurerm_storage_account.storage-account.name}/share -o vers=3.0,dir_mode=0755,file_mode=0755,serverino,username=${data.azurerm_storage_account.storage-account.name},password=${data.azurerm_storage_account.storage-account.primary_access_key}",
    ]
  }
}

Similar it is also possible to inject the mount to /etc/fstab, also by using cloud init.

However, after applying, the key (accessed as primary_access_key) remains in the terraform state. Is this considered safe? Is there a way to delete the key from the state? Also, when using cloud-init, the key will remain in the custom data of the VM.

Is there another safe way to mount data on the VM without storing the keys?

Recently I've read a good book about Terraform that I think answers your question (Terraform Up & Running 2nd edition. ISBN-13: 978-1492046905. ISBN-10: 1492046906).

Basically it says that all data in Terraform state file is stored in plain text, including sensitive data. The author says that at the moment of the writing (May 2019) there is no real solution for this, but workarounds like encryption and IAM to control user access based on policies (the book talks about AWS, so you have to find the Azure equivalent and prevent unauthorized access to the state file).

You can find more details on Chapter 3, pages 78 and 79. There is also an open issue at Github about this: https://github.com/hashicorp/terraform/issues/516.


Jenkins Pipeline - input only if branch matches I want to ask for input but only if we are on a certain branch.  But the input is run before the when. Can this be done with declarative syntax?

For example this will always prompt even when the branch is not master

stage('only on master') {
   when {
      branch 'master'
   }
   input { ... }
}

I see two approaches  :

Save current branch name on a variable and check it in a if statement

CURRENT_BRANCH = '...' // some value or parameter

stage('only on master') {
  if (CURRENT_BRANCH == 'master') {
  input {
    ...
  }
}

// be sure to check if input was set or define a default value

or

Run a shell script to know if current folder is a git repo on master

The following command check if current folder is a git repository  printing "true" ( or returning a error if does not have repo)

git rev-parse --is-inside-work-tree

The next, print the current branch name

git branch | grep \* | cut -d ' ' -f2

Use both to create a script that sets an environment variable.
Check the value on your pipeline, to show the input

stage('only on master') {

  // Set true on IS_ON_MASTER if current branch is master,
  // otherwise set false

  sh ./check-branch-master.sh

  if (env.IS_ON_MASTER) {
  input {
    ...
  }
}


How to manage images versions via docker-compose with Ansible I planned to use docker-compose and a few scripts to manage the deployment of my app on staging and production.
My idea was to use one docker-compose.yml with one .env file for each environment. In those .env files I would have set some specific values and the version of the different images that compose my application.

But then I discovered Ansible and it looks like my few scripts powered by a community of experts. So I started to investigate it and I would like to use it.

Based on the best practices, I can easily imagine having a standard directory layout[1] with the one inventory for each environment. And a couple of roles.

However, I have no idea how to manage the versions; how to describe which version is executed in one environment?

Ideally, I would like to find a simple solution where my CI can update this version.

Thanks

[1] https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html#directory-layout

The best way is of course (like Zeitounator comments) to use a inventory var. Either you use the version number in the docker-compose task as a variable or you create a Jinja2 template docker-compose.yml.j2 and use the variable there. That's what I do.

Example:

all:
  hosts:
    host1.prod:
      image: "xyz/12"
    host2.dev:
      image: "xyz/latest"

And the docker-compose.yml.j2 contains the variable:

version: '2.1'

services:
  myservice:
    image: "{{ image }}"
    ...


Provide customers with access to Azure Container Registry I'm using Azure Container Registry for private Docker images. I want to let around 500 customers pull images and use the Docker V2 HTTP API to read metadata.

Service Principals are one way to go about this. My plan is to create a single Service Principal, and grant it the AcrPull role on the Azure Container Registry. I'd then create a time-limited password credential for each customer.

The customer flow would be something like:

Authenticate using the Service Principal client secret, to get an Azure AD Access Token
Hit the Azure Container Registry OAuth token endpoint, swapping the AAD Access Token for an ACR Access Token and Refresh Token
You can now use docker login using the ACR Refresh Token as the password
You can also access the ACR HTTP API using the ACR token

These steps are not carried out manually, but by automatic scripts running on customer's servers.

After much research (Microsoft's docs here are not great), this approach seems to work as expected.

Is using a Service Principal the "idiomatic" way to approach this? Are there any benefits to creating an individual Service Principal for each customer?

There is a balancing act here between the simplicity and maintainability of your solution, and how auditable creating separate service principals or users for each of your external customers.  I suspect only you can actually decide which is the best practice in your specific legal and regulatory situation.

On the side of having a single Service Principal with multiple passwords, there is a minimal amount of configuration required and other than perhaps limitations imposed by Microsoft on the number of authentications per second or the number of passwords that can be created for a Service Principal it should work quite nicely.  The downside is your resource logs won't give you any information about who pulls images, you will have some logs about issuing a short-lived password, but you can't prove that's related to a docker login.

On the side of creating a Service Principal per client means your resource logs will tell you who pulls images.  However, you will need to automate the creation and maintenance of those Service Principals to account for clients joining and leaving.

Why does it matter? it may not for you, however if one of your customer's machines was compromised and the images leaked onto the internet you may want to hold one of the customers to account.


Does Docker on MacOS support tmpfs? I am new to Docker and I want to run Docker with ram only storage. In other words, I do not want Docker to save anything on persistent storage. I found out that on Linux Docker supports tmpfs. Is there any such support for MacOS? I have tried setting --tmpfs flag and docker run commands run fine on mac with that flag.

According to the docs tmpfs is linux only.

RAM disk

If you want "RAM only storage", you can literally use a RAM disk. ie:

diskutil erasevolume HFS+ 'DockerDisk' `hdiutil attach -nomount ram://2097152 `

Then you can point volumes at /Volumes/DockerDisk. Drop and recreate at will.

Docker will still store its VM and images on disk. See: Docker > Preferences > Disk.


Ansible playbook for configuring runit to supervise an uberjar? We are using Ansible 1.9 for deploying Spring Boot microservices.  We are using CentOS 6 and the services are packaged as rpms. We intend to configure the executable jar files to be supervised by runit, instead of SysVinit or Upstart.

Is it reasonable to package the runit config inside the RPM, and let Ansible only deploy specific properties into /etc/sysconfig, or does a better way to use Ansible for the full runit config exist, so that the service would be decoupled from that host configuration?

So both have their place. On one hand, putting service definitions in the package helps to keep the number of "moving pieces" to a minimum. On the other, it also requires a more complex package build process to at least some degree (like you would have to build your own packages for things instead of using distro packages). If you have the capability to do it in packages, that seems better since moving pieces in prod will always eventually break.


Adding default external network in docker-compose I am trying to learn docker and understanding docker-compose

As I was trying out the external network section:

 networks:
  default:
    external:
      name: my-pre-existing-network

I understand that
'my-pre-existing-network' needs to be created.

Is it possible to create a new default external network from within the compose file itself?

This is more from a learning/understanding perspective. Thanks.

No that is not possible , it is called external because is created from outside the current docker-compose.yml you are working on


What is the difference between DevOps engineer Vs DevOps consultant Vs DevOps architect? What is the difference between DevOps engineer Vs DevOps consultant Vs DevOps architect?

Considering Role wise & Salary wise & Seniority wise & job demand wise, etc.

An architect is somebody who thinks and creates designs.

A consultant is somebody who gets hired for things like "Teach us how to do so and so", or to answer questions like "What's your opinion about this design?".

An engineer is the one who is objective oriented and pragmatic, something along the lines like "How can I get that design (created by the architect, and validated by the consultant) implemented and make it work"?.

Their salaries may vary (there are cheap and expensive ones), same for seniority (there are juniors and seniors, while everybody starts as junior, not all of them ever make it to senior). And job demand is pretty variable (by region, over time, economical factors, depending on hypes, etc).

BTW, it doesn't really matter what the actual "subject" is, it could be "IT", it could be "Road Construction", or it could even be "DevOps" ...

PS: What's in a name (or jobtitle), how about a DevOps BA, as shown in this post ...


Getting permission denied with docker pipeline on jenkins I have jenkins deployed via helm charts on kubernetes cluster
Now i want to be able to run helm to deploy helm charts applications on my kubernetes cluster and have the following in my jenkinsfile

pipeline {
    agent {
        docker { image 'alpine/helm' }
    }
    stages {
        stage('Deploy') {
            steps {
                sh 'helm ls'
            }
        }
    }
}

I also get kubernetes authentication setup on my jenkins so connection to cluster is fine but the error am getting now is the following

+ docker inspect -f . alpine/helm

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/alpine/helm/json: dial unix /var/run/docker.sock: connect: permission denied
[Pipeline] isUnix
[Pipeline] sh
+ docker pull alpine/helm
Using default tag: latest
Warning: failed to get default registry endpoint from daemon (Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/info: dial unix /var/run/docker.sock: connect: permission denied). Using system default: https://index.docker.io/v1/
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.38/images/create?fromImage=alpine%2Fhelm&tag=latest: dial unix /var/run/docker.sock: connect: permission denied
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

What am i missing and how to i get out of this error?

Thanks

UPDATE

below are the additions/changes to default values.yaml file of the stable/jenkins helm chart

agent:
  enabled: true
  image: "joao29a/jnlp-slave-alpine-docker"
  tag: "latest"
  customJenkinsLabels: []
  # name of the secret to be used for image pulling
  imagePullSecretName:
  componentName: "jenkins-slave"
  #privileged: false
  privileged: true

  resources:
    requests:
      cpu: "512m"
      memory: "512Mi"
    limits:
      cpu: "512m"
      memory: "512Mi"
  # You may want to change this to true while testing a new image
  alwaysPullImage: false
  # Controls how agent pods are retained after the Jenkins build completes
  # Possible values: Always, Never, OnFailure
  podRetention: "Never"
  # You can define the volumes that you want to mount for this container
  # Allowed types are: ConfigMap, EmptyDir, HostPath, Nfs, Pod, Secret
  # Configure the attributes as they appear in the corresponding Java class for that type
  # https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes
  # Pod-wide ennvironment, these vars are visible to any container in the agent pod
  envVars: []
  # - name: PATH
  #   value: /usr/local/bin
  volumes:
  # - type: Secret
  #   secretName: mysecret
  #   mountPath: /var/myapp/mysecret
   - type: HostPath
     hostpath: /var/run/docker.sock
     mountPath: /var/run/docker.sock

everything else pretty much same except for ingress hostnames and admin user password...above is most important modification to default values.yaml file

You need to add the jenkins user to the docker group:

# run the following command as root
usermod -aG docker jenkins


Terraform | AWS | vpc_id causing apply to fail I am working on setting up a Cloud9 environment using Terraform, I ran into an issue when it comes to my Terraform apply and specifically:

Error: Error applying plan:

1 error(s) occurred:

* aws_instance.cloud9Test: 1 error(s) occurred:

* aws_instance.cloud9Test: Error launching instance, possible mismatch of Security Group IDs and Names. See AWS Instance docs here: https://terraform.io/docs/providers/aws/r/instance.html.
        AWS Error: Value () for parameter groupId is invalid. The value cannot be empty

Here is my Terraform code that seems to be related and causing the issue.

resource "aws_vpc" "cloud9Test" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "dedicated"

  tags = {
    Name = "cloud9Test"
  }
}

resource "aws_security_group" "cloud9Test" {
    name = "cloud9Test-sg"
    description = "Cloud9 Group"
    vpc_id      = "${aws_vpc.cloud9Test.id}"
}

When I comment out the vpc_id in the security group I don't have the issue anymore but I do want the security group inside my VPC.

How do I fix this?

Note I do not have anything that should overlap causing an issue, this is a fairly new account with only 3 ec2 instances and 1 cloud9 instance all created manually

I'll go out on a limb here and assume that the issue is actually in your "aws_instance" block and is likely the result of how you are assigning the security groups in the "aws_instance".

Copied from the first linked article, in the resolution:

change

security_groups = ["cloud9Test"]

to

vpc_security_group_ids = ["${aws_security_group.cloud9Test.id}"]

Two related articles:

https://github.com/KainosSoftwareLtd/aws-api-gateway-demo/issues/3

https://stackoverflow.com/questions/31569910/terraform-throws-groupname-cannot-be-used-with-the-parameter-subnet-or-vpc-se/34586893#34586893

If this is not on track, please post the "aws_instance" declaration.

For your own sanity I'd also recommend giving the resources names that are more specific to their role, and not identical, for example "cloud9Test_vpc".


Are distributed distributions like NixOS able to replace docker in terms of autoscaling? Autoscaling means that the number of instances increases if the load intensifies. Docker is used in for example Kubernetes in order to autoscale. Are distributed distributions like NixOS suitable candidates for autoscaling as well?

Autoscaling has nothing to do with NixOS or Docker. This is just the feature of cloud backed applications.

However it is true that the best results are achievable using immutable infrastructure, but you do not need neither of these tools to achieve such. Instead you can build your own AMI (using Packer) and then use AWS CodeDeploy to deploy your application on each of the instances.

These tools are orthogonal, so you can use them, but there is no problem with using autoscaling without them.


AWS permissions for Fargate and SSM I'm trying to create some infrastructure for a service I am building on AWS using AWS Fargate.  I'm using SSM as a value store for some of my application configuration, so I need both the regular permissions for Fargate as well as additional permissions for SSM.  However, after banging my head against this particular wall for a while, I've come to the conclusion that I just don't understand AWS IAM in general or this problem in particular, so I'm here for help.

The basis of my IAM code comes from this tutorial; the IAM code is actually not in that tutorial but rather in this file in the github repo linked to that tutorial.  I presume I need to retain that STS permission for something although I'm not entirely sure what.

I've converted the IAM code from the tutorial into a JSON document because I find JSON easier to work with than the Terraform native thing.  Here's what I've come up with.  It doesn't work.  I would like to know why it doesn't work and how to fix it.  Please ELI5 (explain like I'm 5 years old) because I know nothing about this.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters",
        "secretsmanager:GetSecretValue",
        "kms:Decrypt",
        "sts:AssumeRole"
      ],
      "Principal": {
        "Service": ["ecs-tasks.amazonaws.com"]
      }
    }
  ]
}

IAM Role for Fargate has two policies:

The first one describes which service can assume the role and its permissions. In this case it will be the ecs-tasks.amazonaws.com service (= Fargate) that can call sts:AssumeRole to get all the permissions from this Role.
When Fargate assumes the role it gets the permissions specified within, these are the SSM, KMS and SecretsManager permissions.

If you are using CloudFormation you can do something like this:

  FargateRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        # This says that Fargate can assume this role
        Version: '2012-10-17'
        Statement:
        - Action: sts:AssumeRole
          Effect: Allow
          Principal:
            Service: ecs-tasks.amazonaws.com
      Path: /
      Policies:
      - PolicyName: FargatePermissions
        PolicyDocument:
          # This says what permissions will
          # the Fargate container receive
          Statement:
          - Action:
            - ssm:GetParameters
            - secretsmanager:GetSecretValue
            - kms:Decrypt
            Effect: Allow
            Resource: '*'

If you are building the role manually in the console be sure to select in the first step that it's an IAM Role for ECS Task, that will create the equivalent of AssumeRolePolicyDocument from the above code snippet.

Then in the next step add the policy without the sts:AssumeRole:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "ssm:GetParameters",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}

That should create the IAM Role you need. Ideally you should also restrict the access only to specific resources instead of using "Resource": "*" but for start it's easier to just use the wildcard.

Hope that helps :)


What is the difference between modules and workspaces in Terraform I currently use Terraform workspaces to configure different environments like dev, test, prod, and my code works fine. And I believe I can use workspaces to configure different regions as well.

However, I think Terraform modules can do the same thing, right?

If that's so, when should I use workspaces and when should I use modules? Or they are doing a similar thing in different ways?

I think a key difference between Terraform Modules and Workspaces is that modules can be inherited and used by other modules and configurations. Terraform Workspaces are intended to be a collection of configurations that represent a single environment, whereas Modules are components that can be utilized by one or more modules/configurations. I think you could use Modules in a similar way to how you are using Workspaces, but you would be contradicting the intent of the Modules.

For example, say that you want to create a base configuration file for all your company's EC2 instances that allows traffic over SSH. You would create a module that spins up a VM, modifies your firewall to allow SSH, and any other configuration settings that you would want all of your instances to have. You would then have every VM Terraform config inherit this file. This would minimize the amount of code that you would have to write and allow for any changes that you would want across all of your TF configs to be made in one central module.


s3 static hosting not redirecting my request I followed these instructions to redirect:

platform.demo.interos.net

to

platform.demo.interos.io

The instructions use s3 for static site hosting and use the redirect option:  https://aws.amazon.com/premiumsupport/knowledge-center/redirect-domain-route-53/

But when I use the browser, the browser won't actually redirect me from the .net to the .io page, it just hangs. But when I run this:

curl -I platform.demo.interos.net

I get:

HTTP/1.1 301 Moved Permanently
x-amz-id-2: E4VM/vr+1HtO6Q0vvCNL16nxxxxxxFDvvgPW+EjI4g9Zk=
x-amz-request-id: 175E9039358ECD1B
Date: Sun, 28 Jul 2019 19:30:09 GMT
Location: https://platform.demo.interos.io/
Content-Length: 0
Server: AmazonS3

So what's not working here - why doesn't the browser recognize the 301 and do the redirection?

UPDATE:

I now realize it's because in the browser address bar it often adds an https:// in front, and with the protocol in front, it will not redirect. Anyone know why?

For example:

curl -I https://platform.demo.interos.net

will not resolve. but this will resolve:

curl -I platform.demo.interos.net

The blog post you're following is somewhat obsolete, or at least incomplete in today's environment, because S3's web site hosting endpoints don't support HTTPS at all.  The request will time out.

To enable HTTPS on a custom domain pointing to a bucket, you need a certificate issued in the us-east-1 region of Amazon Certificate Manager (or you can purchase one from a third party and load it into ACM there) and then a CloudFront distribution to handle the HTTPS and forward the request to S3.

See How do I use CloudFront to serve a static website hosted on Amazon S3?  Note that these instructions are still applicable even though you aren't, strictly speaking, "hosting a static web site" on S3.  You're still using the static hosting feature and this is how you enable HTTPS.


Halt execution of Bamboo build or deploy, without throwing an exception This seems so trivial, I am almost shy to ask...
Depending on certain conditions, which I am able to check by looking at Bamboo variables, I sometimes want to just stop the execution of a build and/or it's deploy. The possibility to conditionally execute steps seems very limited in Bamboo, and I am having trouble solving this.
I found a workaround, which is running a script that exits with an error, like a powershell script simple as this

exit 1

But that obviously causes the build to appear as failed, which really isn't what I want. I just want to prevent a deploy in certain conditions.

Does anyone know an elegant way to accomplish a full stop in a Bamboo plan?

You can manually stop a build in progress. From Stopping an active build:

To stop an active plan build:

Click Dashboard and then the All Plans tab.
Click the 'Stop' icon next to the active plan you want to stop.

It also appears possible to stop the build/deploy programmatically, using the REST API, more specifically making a DELETE request to the appropriate url:

/queue/{projectKey}-{buildKey}-{buildNumber : ([0-9]+)}?stage&executeAllStages
...
DELETE
Stop build execution.
available response representations:
401
Returned when user does not have sufficient rights to view or execute build for specified plan
204
On success (success is also when build was already completed - so nothing to stop)
404
Returned when specified job does not exist


What is Verdaccio's default login? Back in the day admin:admin worked, but that results in an access denied. verdaccio:verdaccio does not work as well. The Dockerfile does not seem to contain a default password as well.

Command

docker run -it --rm --name verdaccio -p 4873:4873 verdaccio/verdaccio

Log

 http <-- 401, user: null(172.17.0.1), req: 'POST /-/verdaccio/login', error: ba
d username/password, access denied
 http <-- 401, user: null(172.17.0.1), req: 'POST /-/verdaccio/login', error: ba
d username/password, access denied
 http <-- 401, user: null(172.17.0.1), req: 'POST /-/verdaccio/login', error: ba
d username/password, access denied

GitHub

https://github.com/verdaccio/verdaccio/issues/1599

adduser

When http://localhost:4873 is opened, an instruction is shown how to create a new user. However, this fails:

npm adduser --registry http://localhost:4873
Username: (admin) wow
Password: (<default hidden>)
Email: (this IS public) (jane@example.com) admin@admin.admin
npm ERR! code E403
npm ERR! 403 403 Forbidden - PUT http://localhost:4873/-/user/org.couchdb.user:wow
npm ERR! 403 In most cases, you or one of your dependencies are requesting
npm ERR! 403 a package version that is forbidden by your security policy.

Perhaps the company's proxy is causing the issue?

After issuing npm config set https-proxy http://company-proxy and running the npm adduser again, a 403 is still returned.

There is no default password, actually none at all. You must create a new user with

➜ npm adduser --registry  http://localhost:4873/
Username: wow
Password:
Email: (this IS public) verdaccio.does.ignore.email@whatever.com
Logged in as wow on http://localhost:4873/

Then you are logged in.


Is there any motivation to achieve SixSigma or similar in DevOps? According to SixSigma quality level, you would succeed in 0.99966% of all iterations of same process repeated many times.

  A six sigma process is one in which 99.99966% of all opportunities to
  produce some feature of a part are statistically expected to be free
  of defects.

So if I think about build or deployments errors, maybe going as far as SixSigma goes is for time being too much, still with continuous delivery there can be quite many iterations. Does anybody take this into consideration to make "some" sigmas?

This may sound mean but 6 Sigma does not translate that well to computer systems. The general ideas are valid but the numbers are not. It is intended for the meat world, not a place were we routinely do calculations on gigs or tb's of data a day. Assume you run 1 trillion operations (not much when a regular desktop computer can do ~12 billion a second) at 99.99966% success you are still failing 350k times!

Yes, most of the operations on the code side will actually be made up of multiple CPU operations
Yes, some of this is caught by hardware via ECC etc

What would stand up would be trying to use S6 against the changes you make to a computer system (code pushes or otherwise). Not the operation of existing code (which needs a much higher level of reliability than what you'd use to evaluate PPM on a shop floor for physically created stuff).


Dynamically change config files in c# Application based on environments(dev,QA,Prod) through Bamboo I have a windows service written in .net c#. We have three different environments say dev, test and prod. I also have 3 different config files for 3 environments say devAppConfig , TestAppConfig and ProdAppConfig.

Earlier, we used to deploy manual, so we used to replace the config files and deploy the binaries.Now the deployments are to be automated for that we are using Bamboo.

Now my question is how do I dynamically change the AppConfig for different environment deployments.

I have 3 different stages in Bamboo naming DevDeploy , TestDeploy, ProdDeploy. When I run these stages, it has to change the config file and do the deployment, but I'm not sure how.

Can anyone guide me in the right direction for my issue?

Note: I could have added a prebuild event to copy the appropriate config file before building the code. But the main aim here is to able to use the same binaries across all the environments without building it every time.There has to be some setting somewhere (not sure where) to determine which environment it is being deployed and use the appropriate config file.

In your deployment project configuration for each environment you could add a task prior to the actual deployment which would take care of setting up the right configuration file.

It could be a script simply copying/replacing the configuration file just as you currently do manually. The script could be invoked with some pre-configured argument to identify the environment or it could detect the environment via some environment variable configured for each the deployment, see Additional deployment environment options


Docker Swarm join-token not working I am trying to add a new node into my swarm over the internet but it doesn't seem to connect through a proxy.

I am trying to proxy using Caddy. The proxy configuration results in the same as for any other server like proxy_pass in nginx.

Caddy automatically sets up HTTPS for the subdomain and all traffic is set to be redirected from port 80 to 443. My caddy configuration is:

swarm.mydomain.com {
    proxy / http://<docker-swarm-manager-ip>:2377 {
        transparent
    }
}

From the docker swarm manager, I get the join-token which gives me something like:

docker swarm join --token <some-long-hash> <docker-swarm-manager-ip>:2377

On the node I want to connect to the swarm, I have tried the command:

sudo docker swarm join --token <some-long-hash> swarm.mydomain.com

This gives me the error:

  Error response from daemon: rpc error: code = Unavailable desc = grpc: the connection is unavailable

I then tried specifying the SSL port:

sudo docker swarm join --token <some-long-hash> swarm.mydomain.com:443

This gives me the error:

  Error response from daemon: rpc error: code = FailedPrecondition desc = transport: received the unexpected content-type "text/plain; charset=utf-8"

I am new to docker swarm and I can't find any similar setup online. Any help is appreciated.

Note:

I am able to ping the manager from the node successfully.
port 2377 is is open on the manager and the node I want to connect.

The swarm protocol is not http, so you will not be able to pass it through a proxy. Swarm nodes need direct communication between each node on port 2377/tcp (or your configured swarm port), 7946/both, and 4789/udp. If you configure an overlay network with the secure option for IPSec, then you'll also need protocol 50 open between the nodes.


Dedicated/Separate Git Repo as Terraform Remote Backend I see that there are multiple options for Backends, however git isn’t one of them.
I believe git can be used as a backend for tfstate, in the below steps:

Create a dedicated git repo
Commit the tfstate files in a specified directory and branch. Only fetch the required files, rather than the whole repo, as this repo could be used to store multiple tfstate files on different directories.
For locking and unlocking:
Method 1 (git LFS): use ‘git lfs lock’

Method 2 (git - where lfs isn’t possible): Use another file suffixed with .lock, which should contain the lock state.
Is there any plan for Terraform to release git as a backend?
I've also thought of ways to use the HTTP backend to upload to a git, no luck yet.
Thoughts on how this can be achieved?
There's already a Github/Gitlab/Bitbucket provider.
Thanks,
Note: Also asked in the Terraform Discuss

Using git for terraform remote state management is an interesting, but non-standard implementation. Storing binary / large files in git, integrating a locking system and a few other concerns come to mind. Using s3, Buckets or other storage options provide the multiple statefile management (versions in buckets, folders in buckets or terraform folder structure / workspaces) and other functionality you mentioned.
If you would like to pursue using git for remote state management, check out the community project terraform-backend-git. There is also a good discussion around it in the terraform issues section: https://github.com/hashicorp/terraform/issues/16616.
If you attempt to use the tool from the community, it would be great to hear about your experience. Good luck!


How many iops do I actually get in a General Purpose SQL Database in Azure? I'm running a serverless SQL database in azure, taking a load from some data transformed by Databricks.
Every time I try to write to the database the, IOPS percentage shows up as 100% which makes it clear that this is my bottleneck.
In the database configuration tab, I can see that for general purpose I get 500-20000 IOPS provisioned. But how do I actually get and can I control this in any way?

Follow this article
But in serverless mode you will have to look at limits for the vCores your DB is using at that second, so if vCores are 4 then you will have limit of 1280 and Throughput of '5-7 ms (write) and 5-10 ms (read)'


ansible-playbook: how to dynamically, conditionally set a large deployment environment (variable space) in a playbook? I am using an ansible playbook to deploy a php web application in aws.  The deployment is repeated many times, with a different environment, for each customer.  The different scenarios range from single server deployments to full deployments using elastic load balancers, and use varying code repositories based on product type.
To allow a human to deploy this app repetitively and adjust the deployment type and environment, I am running ansible-playbook with --extra-vars "@deployment_vars_file.yml" and copying and filling this vars file on a per-deployment basis, instead of putting the variables in group_vars/all.yml, so that I can keep a copy of the variable space I set for each deployment, and avoid overwriting (or .template suffixing) group_vars/all.yml and multiple roles' vars/main.yml files, for each deployment.  I rarely rely on variable precedence, since the role and play variables inherit (are constructed out of) my deployment vars file variables.
My roles are already run conditionally at the play level, and I have a handle on how to run tasks/blocks/roles/plays conditionally, depending on the environment type that is defined in the --extra-vars file.
I need to define many other variables, dynamically (during the playbook run), conditionally (based on the values of variables in my --extra-vars file), with as little work as possible (i.e., in one place), at the beginning of the playbook.  My roles' vars/main.yml files, as well as vars: play keywords, will inherit these other variables, (in addition to directly inheriting some of the --extra-vars variables).  The values of these other variables need to persist at the playbook level, not just within each play. I don't want to have to do this in multiple locations with the same lines of code.
How can I do this, and what is the most simple way?

Yes, an input host file for ansible looks like https://pastebin.com/Vtfw5FtV
The upper part node stuff is generated using terraform, we can dynamically up/down scale the number of nodes with terraform and terraform outputs the yaml used for the hosts part of the file. The tenant-specifiek yml vars (about 150 additional lines of tenant specific config yml) are kept in a separate git location and a jenkins step/python concatenates them together and pushes them to the host directory location for each tenant and runs individual roles (hotfix) or all of 'em (full deploy) or perhaps just stop/start. The Jenkins job is pretty much a control screen with a lot of dropdowns, radio buttons and selectors. We can select how many nodes we want for a single service, which build to grab from nexus, do a full deploy, a single service hotfix or just a configuration hotfix. All this is selectable/executable per environment, per tenant and per individual module/service. The Jenkins screen is dynamically build from the host (targets) & roles (actions) directory.
Again, we're most likely abusing Ansible "best practices" a fair bit but for us this approach has been working remarkably well for last >3 years in terms of overall manageability and gives us maximum flexibility of the whole CI/CD pipeline.


How to use port 5433 with Azure managed postgres We are looking to use Azure managed postgres but I am having some issues with connecting on port 5433 I can connect on port 5432 fine. I need just port 5433 to work, is there a way to access that port?

You can not change the port that Azure Database for PostgreSQL server uses. Connections to your Azure Database for PostgreSQL server communicate over port 5432. When you try to connect from within a corporate network, outbound traffic over port 5432 might not be allowed by your network's firewall. If so, you can't connect to your server unless your IT department opens port 5432.

If you truly need to use port 5433 you can install PostgreSQL on an Azure VM (IaaS).


VPN container providing access to host's LAN? I'm kinda new to this Docker thing and I'm interested in getting a biffed up VPS for some containers (websites/data, mysql, nginx, rocket.chat, etc) but I wouldn't like to access them in the wild (even through SSH) and some VPS hosts's console access are plain cumbersome. So, here's my question:

I've got around 3 separate VPS sharing a common network: 10.0.0.1 thru 10. I'd like to know if, in the host (10.0.0.5) I can set up a docker server, then, inside I can set up an OpenVPN container (or any other kind of VPN) so I can connect from outside to the other VPSes (10.0.0.3, 10.0.0.8, and so on).

Bear in mind that I'm not looking for a privacy-related VPN, I don't want to route traffic nor go outside with the VPS's IP, I just want to "get inside" the internal network I have shared with my VPS (Vultr in case you're wondering, they have a private IPv4 service only for the VMs in my account so it's safe for them) through a VPN for added security and commodity, so I can access a few hosts naturally (and maybe containers as well?) using SSH/RDP/etc. with a single connection instead of having 10 profiles in PuTTY for each server.

Any pointers would be greatly appreciated. Please also take note I'm a newbie regarding docker, I'm a developer & IT guy so I know my way around, but I don't have much idea about networking so I'm not sure if it's possible - and if it is, then can you please tell me how?

Thanks!

I am accessing my home network with a VPN running in a docker container in my home server.

That sounds similar to what you want to do.

Here's how I configured it (using this docker image - note that the documentation of the docker image should be enough)

Use a “convenience” environment variable to store the path to your persistent storage location that will be bind-mounted to the container.

OVPN_DATA="/n7wings/openvpn/"

Run an ephemeral instance (–rm) of the image to initialize the data directory of the container (ovpn-host should be the hostname of your openvpn server)

docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig -u udp://ovpn-host

Run an interactive ephemeral instance of the image to generate the opevnpn CA certificate and server key (you will have to type your passphrase for the private key)

docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn ovpn_initpki

Run the VPN service: start and detach the container (-d) and map a host port to the UDP container port where the openvpn server process is listening (1194). In this example the host port will be 1195

docker run -v $OVPN_DATA:/etc/openvpn -d -p 1195:1194/udp --cap-add=NET_ADMIN kylemanna/openvpn

Generate client configuration (i.e., add a user to the VPN). If you omit the nopass option, the client key will be encrypted with a passphrase.

docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn easyrsa build-client-full vince nopass

The client key will be in ${OVPN_DATA}/pki/private and the certificate in ${OVPN_DATA}/pki/issued
Retrieve the client configuration to a local file:

docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_getclient vince > vince.ovpn

If you need to add more users, just repeat the last two steps to create a user configuration on the server and retrieve the ovpn file.

Full reference: https://thealarmclocksixam.com/2017/03/07/setup-your-vpn-in-docker-with-openvpn-in-5-minutes/


How to evangelize DevOps and tools in a low acceptance environment? An evocative learning about innovative disruptive technology is that you might find out that not everybody is going to be excited about tools which can give you more productivity.

To make this question less generic, what are do's and dont's - so as they also say, lessons learned - of evangelizing Docker in your organization based on real experiences with different roles?

This is not docker-specific, but this general rule for evangelism applies: different audiences require different evidence.  In general, software developers (and managers from a development background) want to see it in action, so POC's with measurable outcomes are preferred.  Other disciplines and executive stakeholders may be OK with case studies and presentations before giving the greenlight; your goal is to identify which audience you're talking to, and begin to build a comprehensive argument.

Start small; find one key stakeholder, and present your case using the evidence that best suits them.  Be patient; change in mature organizations is usually slow, so getting a small project up and running is a major accomplishment.


Jenkins connect to Windows 10 agent I have a Jenkins master on a Linux machine and I would like it to be able to connect to a Windows 10 agent, how do I do that?
I can't find much in the official documentation about agents in general, and as for searching the web, the results show how to do it using JNLP, which only works on Java 8 and not on 11, and I don't want to downgrade from 11 to 8.
I've been able to set up Linux agents, to which I connect using ssh (like this), but the same process doesn't seem to work for the Windows agent.
I am at wits' end here, do you know or have any up-to-date resources on connecting to Windows agents? Thanks.

Have you checked the following link: https://wiki.jenkins.io/display/JENKINS/Step+by+step+guide+to+set+up+master+and+agent+machines+on+Windows
I advise you to configure a node on the master ( manage --> manage nodes ) and set the launch method to  launch agent by connecting it to the master. After that you have three methods to connect the Windows agent to Jenkins master, the easiest way will be to lanch it from a browser.

This should work enven with jdk 11.


Is it possible to show a code coverage badge in bitbucket repositories? In github it is possible to show a code coverage button. Is this possible in bitbucket as well?

https://github.com/dwyl/repo-badges

Attempts

This site was found https://bitbucket.org/site/master/issues/12797/provide-project-status-build-status-badges

Navigating to https://bitbucket.org/<user>/<project>/addon/pipelines/home#!/badge/branch/<master>?svg=true returns a dead link

You can certainly show a code coverage badge in a bitbucket repository, but it would have to be from an external source like jenkins, codeclimate, etc.

According to your link, if you are using pipelines they now show a code coverage badge on the project overview page. If you want to add that to your README or display it somewhere external, that functionality has not been implemented.


How to promote a build when certain value in output is above threshold? I'm using the Promoted Builds Plugin and I'd like to promote my freestyle job to have a gold star when a certain value in my output is above a given threshold (e.g. above 1.00).

Here is a sample output from backtesting jobs:

Pass  Profit Total trades Profit factor Expected Payoff Drawdown $ Drawdown %
   1  622.71           76          1.38            8.19     526.77       4.94
  10  301.09           36          1.38            8.36     526.77       4.93
  14  240.96           62          1.18            3.89     589.60       5.60

My aim is to promote the build to gold when the first line right after the header line has a Profit factor column value above 1.00 and only to silver star when it is below.

How can this be achieved?

What you are looking for is at the bottom of the GitHub README describing Job DSL for the Promoted Builds Plugin. Link to GitHub is on every plugin's page. There are also links to plugins it depends on and you should read the documentation for those like the Job DSL plugin wiki. You can also ask questions in the Job DSL forum or simply read the documentation. Here is a plugin that extends the promotion syntax of the DSL.


Configure VPC for ECR access I am creating my infrastructure with Ansible. I create a VPC and an ECS cluster where I use Fargate to run my docker containers.
When the tasks are started I am prompted that the task was not able to pull the Image from the ECR.
STOPPED (CannotPullContainerError: Error response from daem)

I know for a fact that the URI to the image and the IAM permissions are correct. Also I don't think the issue is in the AMI or docker image, because when I create the same scenario over the AWS UI, it works fine.

Therefore, I assume my VPC is not configured correctly in a way that it can not connect to the ECR.

This is how I create my VPC and Subnet:

- name: "2) Create VPC"
  ec2_vpc_net:
    name: "{{vpc_name}}"
    state: present
    cidr_block: 10.10.0.0/16
    region: "{{ region_name }}"
  register: vpc_net

- name: "3) Create VPC Subnet"
  ec2_vpc_subnet:
    state: present
    map_public: yes
    vpc_id: "{{vpc_net.vpc.id}}"
    cidr: 10.10.0.0/24
    tags:
      name: "test-subnet"
  register: vpc_subnet

How can I ensure my VPC and subnet is configured correctly so that the docker daemon can connect and pull the image from the ECR repo?
Am I missing an important attribute?

This error CannotPullContainerError usually occurs when you have no access to the Internet from ECS and thus it doesn't able to pull an image from registry.

Make sure you have networking configured in such a way to have an access to Internet https://stackoverflow.com/questions/48226547/aws-fargate-cannotpullcontainererror-500

Manually you can confirm everything with such steps

Find route table. Check whether it belongs
to your VPC.
Find proper NAT (you need NAT Gateway ID) (IMPORTANT: NAT should be launched in PUBLIC
subnet)
Create a route for that

$ aws ec2 create-route --route-table-id "rtb-037148d7b5967a231" --destination-cidr-block "0.0.0.0/0" --nat-gateway-id "nat-0c137ae8a2b409088"

You can also use internet gateway instead of NAT but the main idea is the same - to create connectivity from service to registry.
If you have that in place and verified you can easily rewrite that to ansible tasks.

Update
To start an ECS task you want to use ecs_task module https://docs.ansible.com/ansible/latest/modules/ecs_task_module.html

To pull image from repository and start it you can use these tasks

- name: docker login
  shell: "$(aws ecr get-login --no-include-email --region {{ default_region }})"
  args:
    executable: "/bin/bash"

- name: pull latest app image
  docker_image:
    name: "{{ ecr_repository }}/myapp:{{ image_tag }}"
    force: yes

- name: run application with docker compose
  docker_service:  # module is called docker_service before Ansible 2.8
    project_name: "myapp"
    project_src: "{{ app_dir }}"  # path to a directory containing a docker-compose.yml
    pull: yes
    recreate: always  # run scheduler from a clean state
    remove_orphans: yes  # remove containers for services not defined in the compose file
    state: present  # specifying present is the same as running docker-compose up
    restarted: yes  # use with state present to restart all containers
  register: output

but in case of Fargate the above is not needed as everything is handled by task definition


How to get the next friendly name that Docker will assign to the container it will make? Background

When you start a Docker container as in the following example:

$ docker run -ti ubuntu:latest

Docker docker automatically assigns the container an ID and a friendly name.

Docker friendly names are of the format adjective_famousPerson.

Problem

I would like to get the friendly name that Docker will generate for the next container, so that I can assign that name to the hostname parameter of my docker run command.

For example:

$ docker run -ti --hostname docker.next_friendly_name ubuntu:latest

Question

Is there a built in way to do this with docker?

I need a friendly name that is randomly generated, but I don't want to script it up myself if there is a way to do this with Docker or even Docker Swarm or Kubernetes.

Additionally, the name of the container and the hostname of the container need to match.

As the name of the container is randomly generated you cannot get it before it's generated (by definition). You can use this service to generate random names :

NAME=$(curl -s https://frightanic.com/goodies_content/docker-names.php); \
docker run -ti --name $NAME --hostname $NAME ubuntu:latest


Trick a docker containers DNS to resolve a different url I am not sure I know the terminology to describe this correctly, my apologies.

But for a contrived example. Say I have a docker container that runs code that accesses "google.com". But I want all requests that the container sets to "google.com" to actually go to "facebook.com". In other words I want the DNS lookup of one domain name to use another domain when reloving. Is there a way to do this with docker or docker-compose?

My ask is a bit more than this, because I know that often multiple websites are hosted on the same server (same IP), so I need a way for the conversion to happen at the domain name level, not at the IP level.

Thank you

If you want a DNS lookup of one domain to resolve to another one, you have 2 options.

Modify the Container /etc/hosts file. By default any DNS lookup is done there. You schould do that in the Dockerfile.

Modify /etc/hosts in a Dockerfile

Maintain your own DNS servers with all mapping records. On container runtime just pass the DNS server.

Docker docs

  --dns   The IP address of a DNS server.


How to integrate multiple "feature" react apps into monolithic PHP repo? We have a mono-laravel-repo and are in the process of migrating to a cleaner, easier to maintain dev reboot.

We have recently moved towards migrating our existing PHP based views into multiple React-based "features" that live in a /react directory and are organised into /react/featureName sub-directories. This means Laravel's controllers mostly handle data and authentication only.

Each feature is maintained by one or more developers, have their own dependencies, tests and functionality.

We're currently serving builds in the public root of our project organised by feature:

js/FeatureName.hashed.js
css/FeatureName.hased.css

which works great except that we share many packages between features. React itself, react-router etc.

We intend to use Laravel's mix.js as much as possible, but hacking directly on webpack.config is also an option.

I've explored the mix.extract() option that allows us to define vendor.js builds separate from other application logic but it stops there.

So here are my question(s)

What approaches are there to manage multiple package.json files that live in each feature directory
How do we identify common packages and lock them into a root /package.json file. (Independently, all developers already use yarn)
How to accomplish this with as little friction as possible for all react based developers - so that they get to simply continue with whatever they are already doing and DevOps handles the gritty bits?

I understand there might be a need for more information, please shout out what you need in the comments and I'd be happy to update the question.

What approaches are there to manage multiple package.json files that live in each feature directory

One could leave the package.json in every feature directory and let the CI read the package.json when building the app. One could define the versions of dependencies in the package.json to get control about the app. If the app works with version A.B.C of a certain library then one could define this version to prevent that an update will break the app.

  How do we identify common packages and lock them into a root /package.json file. (Independently, all developers already use yarn)

Discuss this with the team and figure out what is the best solution for everybody. Multiple solutions are possible. Everybody should be happy with a certain approach.

  How to accomplish this with as little friction as possible for all react based developers - so that they get to simply continue with
  whatever they are already doing and DevOps handles the gritty bits?

DevOps in my opinion means that a certain team that consists of multiple specialties, e.g. test, dev, ops is responsible for a certain app. This means that the whole team should agree with a certain approach and everybody should work and handle issues according that process instead of DevOps handles the gritty bits which is about silos and "throw it over the wall"


What is special about the admin in chef? I've read over the chef docs for a while now and I can't seem to find anything that says what an admin does in chef.  I can do all sorts of things with making people admins, but there's no indication of what this does for me.  Is there a clear answer somewhere?

A Chef admin can create, modify and delete other users and clients. Basically an admin can manage access to Chef.


How can you pass private environment variables to FARGATE tasks specified from a Docker Compose config I am using an existing docker-compose.yml to migrate an app to ECS. The config references several $ENV vars which I used to pass in simply as environment variables when running docker-compose up. One of these is a password, which used to be specified on the Docker host. In ECS you are advised not to pass sensitive stuff along with the env-vars, so I'd like to improve this and use the best practises.

In FARGATE, I can't see how to pass these secure parameters into to the ecs-cli compose service up without sending them into the docker-compose used by ecs-cli (and then they end up displayed in the console, I guess possibly logged, etc).

The ECS CLI documentation explains to use SSM

  secrets allows you to specify secrets which will be retrieved from
  SSM Parameter Store. See the ECS Docs for more information,
  including how reference AWS Secrets Managers secrets from SSM
  Parameter Store. value_from is the SSM Parameter ARN or name (if the
  parameter is in the same region as your ECS Task). name is the name of
  the environment variable in which the secret will be stored.

But under the ECS Docs

  "This feature is not yet supported for tasks using the Fargate launch
  type."

I presume the "yet" means there will be one day.

I wonder if there's an current alternative for Fargate, or if we have to be less secure while they add support for it?

This is a well-written and up-to-date article which summarises the problem I've hit and confirms there isn't really a perfect solution at the moment in Fargate.

https://hackernoon.com/secrets-management-within-aws-ecs-1b6975819ccd - Connor Beardsmore

The alternatives listed are:

Bake into Docker - not an option with an open source repo
ECS Environment Variables - as I mentioned less secure
Docker Fetches from SSM Parameter Store - secure but "architecturally dirty"
Wait for Fargate SSM to be able to inject sensitive data!

Re. option 2. the problem is the environment variables are visible in the AWS console. But I wonder: if an attacker has access to your AWS console, you're pretty screwed already? So isn't this in fact an practically acceptable solution?


Why to overrule role vars? There is quite a simple setup I am sure many of us have come across:

You have lean, slim and dedicate roles
Every role has its own role vars
Playbooks include the roles

Let's assume you have a base path for an application, most likely you will need this path in many roles. To keep the roles independent you would define this base path in every role vars where needed.

Now you include some roles in a Playbook and in case of a change on e.g. the base path you need to change that in every role.

One help would be to use extra vars to overrule the role vars, but what if there are many potential candidates for a change?

So I would like to be able to include_vars in a playbook to overrule the variables in the roles. But this is not possible as:

include_vars is not allowed in Playbooks
var_files (what would work in Playbooks) will be over written by role vars

Question: how do you do solve this situation?  Maybe I want something odd and there is a much simpler way?

Variables have different priorities, see vars precedence.

As of Ansible 2.x:

  role defaults
  inventory file or script group vars
  inventory group_vars/all
  playbook group_vars/all
  inventory group_vars/*
  playbook group_vars/*
  inventory file or script host vars
  inventory host_vars/*
  playbook host_vars/*
  host facts
  play vars
  play vars_prompt
  play vars_files
  role vars (defined in role/vars/main.yml)
  block vars (only for tasks in block)
  task vars (only for the task)
  role (and include_role) params
  include params
  include_vars
  set_facts / registered vars
  extra vars (always win precedence)

For roles you would generally use myrole/defaults/main.yml for easily overridable vars (e.g. installation path or smth) and use myrole/vars/main.yml for vars that you don't want to be easily overridden (smth like required packages).


Unable to parse YAML: mapping values are not allowed here Good morning. I'm migrating from Gitlab to CircleCI and try to map out a simple pipeline ("workflow") that basically should look like this:

[build] -> [tests] -> [release*]
* only on tags

I wasted endless hours literally copy-pasting example configs from the documentation. I defined three jobs:

build
tests
release

And I created a minimal workflow that just echos the job title, includes some requires (to map the pipeline) and have a condition that only triggers releases on tags.

However, I always get a YAML parsing error:

#!/bin/sh -eo pipefail
# Unable to parse YAML
# mapping values are not allowed here
#  in 'string', line 25, column 19:
#               requires:
#                       ^
#
# -------
# Warning: This configuration was auto-generated to show you the message above.
# Don't rerun this job. Rerunning will have no effect.
false

Exited with code 1

This is my config.yaml that currently fails:

version: 2
jobs:
  build:
    steps:
      - run:
          name: Compile Binaries
          command: echo "Compile Binaries"
  tests:
    steps:
      - run:
          name: Run All Tests
          command: echo "Run All Tests"
  build:
    steps:
      - run:
          name: Release Artifacts
          command: echo "Release Artifacts"

workflows:
  version: 2
  minimal:
    jobs:
      - build
      - tests
          requires:
            - build
      - release
          requires:
            - tests
          filters:
            tags:
              only: /^v.*$/

I also tried online YAML validators but they always say this is valid YAML, so something is wrong with the way Circle expects the config to be. What could it be?

Edit: I managed to even minify the example that causes the exact same error:

jobs:
  build:
    steps:
      - run:
          name: Release Artifacts
          command: echo "Release Artifacts"

workflows:
  minimal:
    jobs:
      - release
          filters:
            tags:
              only: /^v.*$/

Causes:

#!/bin/sh -eo pipefail
# Unable to parse YAML
# mapping values are not allowed here
#  in 'string', line 12, column 18:
#               filters:
#                      ^
#
# -------
# Warning: This configuration was auto-generated to show you the message above.
# Don't rerun this job. Rerunning will have no effect.
false

Exited with code 1

You simply have an incorrect yaml syntax in your workflows jobs list.

From what I could very quickly understand on circleCI's documentation, jobs names are accepted as a shorthand if you don't define any parameters. If you have parameters, the list element is a hashmap with top element being the job name. In your specific case, this should give (note the column after the names of jobs with params):

workflows:
  version: 2
  minimal:
    jobs:
      - build
      - tests:
          requires:
            - build
      - release:
          requires:
            - tests
          filters:
            tags:
              only: /^v.*$/


What is the need of plugins in Jenkins (Sonarqube) On learning about integrating Sonarqube to CI/CD pipeline, I came across the Sonarqube Jenkins plugin which most of the online blogs recommends.

After configuring it, I realized that the working is the same as that of running the sonar-scanner through the command line using below command.

sonar-scanner.bat -D"sonar.projectKey=test" -D"sonar.sources=." -D"sonar.host.url=http://localhost:9000" -D"sonar.login=erwerwerwerwer"

Same is applicable for any tool, as most can be executed through command line.

So I am confused about,

  what's the advantage of using a plugin over just running the tools as
  windows batch command in Jenkins build step.?

Any CI/CD plugin can ultimately be scripted.  However, using plugins gives you better centralized control.

Consider the following scenario: Your SonarQube server location has to move to a new location.

If you are using a script like the example you provided above, you would need to manually update every script.  This process is error prone as you may miss a script or worse, type the wrong address.  Instead, if you are using a plugin you can update the server location once and be done.