How to capture tools release to trigger upgrade? We capture our applications release by using commits from source code management, new package into the artifactory system etc.,

How can we capture my tools release from vendor so that I will trigger my upgrade script to upgrade my tools itself, for ex: upgrading Jenkins whenever new release available from Jenkins itself?

List of tools am expecting to upgrade:

1. Jenkins
2. GitHub Enterprise
3. Atlassian Jira
4. Atlassian Bamboo
5. Atlassian BitBucket Server

Basically, continuous deployment(with proper pipeline including testing) to production without human interaction for these tools.

Note: I am looking for some example, my requirement is for more than 10 tools. So for Jenkins, it will be Linux and Ubuntu.

From your list of tools:

Jenkins and Github have apt/yum repositories you can use to install and upgrade.

For Atlassian products, best up to date information about repositories I can found is This ShipIt experiment and it seems unlikely to change from this forum post
The workaround may be parsing the mailing list to update the deploy.

Now if you really want something automated to handle the installation and configuration of your tools, you can try a Configuration Management System like Chef/Puppet/Ansible/Salt.

For Chef I know better, there's cookbooks for Jira and Confluence maintained.
For the rest you'll be on your own writing the bits to deploy and configure.

This means having 4 CI/CD systems (at least), two 'groups' validating each other, as a pipeline won't be able to validate and upgrade itself properly (inception). That sounds overkill in my opinion.

At the end of the day, there's no silver bullet for all and every software you may use, each will have its own particularity and way to be installed and you'll have to workaround those bits.


Cross-Platform Deployment Tasks in Bamboo Currently most of my deployments are using a dedicated agent and using Robocopy (windows) or SSH (linux/osx).  A common scenario for my deployments are copying .dlls from a .net project to a server.

I am trying to consolidate capabilities to allow either the windows or linux/osx agents to deploy (using Bamboo).

Is there a simple shared protocol to copy files regardless of the OS or am I correct in assuming I need to setup a solution like OpenSSH on the Windows environment?

There isn't a common protocol that exists on all the platforms for deployment.  However, if you get an SSH server onto the windows platforms then you can use the same mechanisms in Bamboo to do the deployment, and potentially even use the same keys to secure it all.
Windows SSHD - There are a few alternatives (https://serverfault.com/questions/8411/what-is-a-good-ssh-server-to-use-on-windows).  Freesshd you can get administrator level access - been successfully using it with thousands of machines as old as WinXP for years.
The Bamboo SCP task can be used to do the staging to the remote servers - once you have SSH remote access to all your targets.

You can use the Bamboo SCP task to upload files from Bamboo directly to a remote server as part of a Bamboo job. The SCP task is able to copy multiple files and preserves the directory structure for the copied files.

And Bamboo SSH task can then do post deployment install steps, running scripts relevant to the platform.

You can use the Bamboo SSH task to execute a SSH command on a remote computer as part of a Bamboo job.


Difference between chroot and Docker I don't understand the difference between docker and chroot. Yes its nice in terms of the packaging the registry. But somehow I get the feeling its just chroot with extra bells and whistles.

I know I'm missing something. It'd would be great to know how they are different and the need of docker if chroot could do something similar.

I couldn't find this Chroot Vs Docker clear enough either.

Well, the extra bells and whistles is called process isolation, a container gets its own namespace from the host kernel, that means the program in the container can't try to read kernel memory or eat more RAM than allowed.

It also isolate network stacks, so two process can listen on port 8080 for exemple,
you'll have to handle the routing at host level, there's no magic here, but this allow handling the routing at one place and avoid modifying the process configuration to listen to a free port.

Secondly a chroot is still read/write, any change is permanent, a docker container using aufs will start from a clean filesystem each time you launch the container (changes are kept if you stop/start it IIRC).

So while a container may be thought of as process namespace + chroot, the reality is a little more complex.


RDS DB migrations using ECS and Fargate I run my app with Docker and ECS. How can I handle database migrations? With fargate there doesn’t seem to be a way to SSH into the container to run any commands so I can’t run my RDS MySQL migrations by going into the container. I have some migrations to run at times, not for all deploys. Any advice?

I used to run my docker container in a EC2 box so I had the option to run a command inside the container once the container is up. I have migrated to ECS and Fargate now and I don't know how to handle it now.

If you already have the script/tool in place to execute the database migrations then what you might need is to run the tool/script once. AWS ECS allows to schedule the executions of tasks (containers), you can set it up in a way that the container can run just only once.

These ECS tasks can be scheduled via a cron-like job setup or on a CloudWatch event response:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html


No Kubernetes Option in Docker Desktop for Windows Just getting started on Docker and after installing Docker Desktop on a Windows 10 Pro machine, the option to enable Kubernetes is missing from the context menu when using Windows containers. See the attached image. When I switch to Linux containers the option appears. Various reading I've done seems to indicate that it should be available to Windows containers too. What am I missing?

Windows 10 Pro (10.0.17134.1006)

Docker Desktop 2.1.0.5

After update:

Settings after update:

You need to first switch to Edge version by going to the

Docker -> Settings -> General tab.

Then, you have to enable the kubernetes.

  Open Docker for Windows settings by accessing your system tray and navigate to the Kubernetes tab. Select the checkbox for Enable Kubernetes to start a single-node cluster when starting Docker.

Reference: https://medium.com/@damiannolan/kubernetes-on-docker-for-windows-5ca0c6395668

As per this old link and a latest one, Kubernetes needs a Linux master node and thus the kubernetes cluster can be initialized only from linux containers in Docker desktop for Windows. After enabling and starting the docker containers, you can add Windows containers the the running kubernetes cluster.

This will give you much clear context.


CI/CD pipeline with manual advancement? I am currently learning about CI/CD pipelines for DevOps. I have some practice implementing them in some basic scenarios, but I've recently encountered a slightly more complex scenario and I'm not sure the best way to integrate CI/CD pipelines into this project.
The Scenario
The client and the development team have agreed to work in 2 week sprints. Once the sprint has ended, the client's management team wants 1 week to play with the sprint delivery in a sandbox environment (we'll call it UAT) before it is released to production (PROD). This means that releases to PROD happen on "off weeks" from a sprint perspective, if they happen at all for that sprint. If the customer is unhappy with something they'll ask us to not deploy to prod until the issue is resolved. We still continue working in sprints, we just don't release that sprint to PROD.
Question
Clearly the naive CI/CD pipeline approach doesn't work here since most code changes aren't expected to make it straight down a pipeline to PROD. In light of this, what is the best approach here for a team that is still interested in utilizing pipelines as much as possible?
Some Thoughts

We could use a single pipeline with manual approval steps prior to the UAT portion and then again prior the PROD portion. This has the benefit of being a single pipeline, so a single build artifact progresses through the entire pipeline, which seems to be a best practice. The problem is that most things that enter the pipeline are going to get hung up at a manual approval step. Most pipeline runs will then either be stalled or marked as failed.

We could use 2 pipelines. One is totally CI/CD with DEV and STAGING environments. Every two weeks, a separate pipeline would be manually started (or started by some other release mechanism associated with SCM like tagging or merging). This would be in charge of rebuilding, retesting, and deploying to UAT and PROD.

If it's relevant, jenkins would be the pipeline of choice.

The "D" in CI/CD means different things to different people, either delivery or deployment. For both groups, it rarely means release everything immediately to production. For the delivery group, the output of CI/CD is an artifact that can be deployed. For the deployment group, there are typically gates in pipelines with either automatic or manual checks before deployment happens. Automatic gates include the unit tests, CI tests, but also things like canary releases with monitoring of key metrics.
Manual approvals in Jenkins involve adding an input step to your pipeline. E.g.
input([message: "Ready to continue?"])

In my experience, having various stalled pipelines waiting for user approval is fairly common. I often include timeouts to clear out stale prompts, and sometimes I configure old prompts to automatically cancel with a more recent pipeline is approved to reduce the workload on users. If you do the latter, you may want to have a separate pipeline for backout/revert jobs since that would otherwise being a new job that gets promoted to production in an emergency and aborts all the other work in progress.
I still like to separate my builds from deploys with separate pipelines. It gives developers better feedback with their CI pipelines of what's working. Then the deployment team picks from a list of automatically queued up deployment jobs that were automatically triggered from those successful CI jobs.


Cheapest and easiest way to run a periodic Docker process I have a dockerized Python process that a client would like to run once or twice a month. It calls a number of Google APIs and stores the results in a Google Sheet. It does not accept network requests. I want to provide one "magic button" that will spin up the Docker container and then shut everything down.

Some background: I had previously deployed this to an AWS EC2 micro instance via Docker Cloud. My client got a Docker Cloud account and was able to run the container on demand, relatively painlessly. There are two problems with this workflow:

The provisioned EC2 instance (Docker host) was running 24/7, whether
the Docker container was run up or not. This became expensive.
Docker cloud shut down a month ago.

Alternatives seem to be Google and AWS Kubernetes services. My reservation is that they will be too complicated for my client to use. Advice?

AWS offers fargate now, which is managed by them rather than being a service like ECS which you ran the servers for and they manage the control plane. Fargate is basically "serverless" ECS. You can use a cloudwatch scheduled even to run something occasionally on a schedule on a fargate ECS cluster. Therefore you're only paying for what you're using. Something to consider is fargate is quite a bit more expensive than ECS running on your own EC2 instance, but if you're running in frequently you'll likely see a pretty marginal savings.

Another option would be to automate something similar. So create an ECS cluster with an ASG backing it, and adjust scale to match ECS demand, and then use cloudwatch to launch the container. ECS can then scale up and down, but this can be tricky because this method is usually used to autoscale ECS clusters based on cloutwatch triggers, so you might have to hack around in cloudwatch and possibly hack together a lambda script, since you're technically not using it the way it's intended. In the end this would probably be the most cost effective, but in your case the cost savings could be negligible.

In my opinion Kubernetes is a ten ton hammer that you'd be using to hang a picture on the wall. It's way overkill.

You can explore those 2 options and see if they are cost effective, and easy enough to manage. Other container orchestration is usually going to be overkill, ECS has the advantage(usually a disadvantage) of being stupid simple and you pay very little and have basically no operational overhead for the control plane.


docker: Error response from daemon: Conflict. The container name "/basexhttp" is already in use by container Am I even correctly following the documentation on creating and running a docker container below?
[root@ ~]#
[root@ ~]# docker run -d \
>     --name basexhttp \
>     --publish 1984:1984 \
>     --publish 8984:8984 \
>     --volume "$HOME/basex/data":/srv/basex/data \
>     --volume "$HOME/basex/repo":/srv/basex/repo \
>     --volume "$HOME/basex/webapp":/srv/basex/webapp \
>     basex/basexhttp:latest
4b827c082bebe45b8be5b7c956f1ada92407181aeec82dc5f71a63f2c5af537e
[root@ ~]#
[root@ ~]# docker run -ti \
>     --name basexhttp \
>     --publish 1984:1984 \
>     --publish 8984:8984 \
>     --volume "$(pwd)/basex/data":/srv/basex/data \
>     basex/basexhttp:latest
docker: Error response from daemon: Conflict. The container name "/basexhttp" is already in use by container "4b827c082bebe45b8be5b7c956f1ada92407181aeec82dc5f71a63f2c5af537e". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
[root@ ~]#
[root@ ~]#
[root@ ~]# docker container list
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@ ~]#
[root@ ~]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@ ~]#
[root@ ~]# docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED              STATUS                          PORTS               NAMES
4b827c082beb        basex/basexhttp:latest   "/usr/local/bin/mvn-…"   About a minute ago   Exited (1) About a minute ago                       basexhttp
[root@ ~]#

referencing:
https://hub.docker.com/r/basex/basexhttp/
https://docs.basex.org/index.php?title=Docker&mobileaction=toggle_view_desktop
where I've copied/pasted what I think are the relevant commands.

You're attempting to create two different containers, each with the same name. Each time you execute docker run it will create a container, and you've named the containers the same with --name basexhttp. These names must be unique, even among stopped containers. To delete an unneeded container, you can stop and remove it with:
docker container stop basexhttp
docker container rm basexhttp

(Note that docker stop and docker rm also work, but it's easier to read and understand the more verbose syntax)


How to deploy docker-compose to a remote host I am new to the docker ecosystem and I am trying to figure out a proper deployment infrastructure for a system I am working on.

Imagine a following scenario: a system consists of three microservices - DB, core server, and an app (server + client). For each of these I created a docker image. I also linked them together with docker compose to describe the system as a whole and described links between the microservices. During a release of the microservice I tag the image with a proper tag and push it to a private docker repository. So far so good, it works locally.

Now, imagine I have three different production clients, and each has it’s own one rack linux server. I am not sure how should I approach deployment of the system with docker:

1) I want to be able to specify different versions of microservices for different prodction servers. How should I approach this? Should I have one general docker-compose file or one file with specified versions for each production environment?

Where should I keep docker-compose? In a repository? How it is copied then to a remote host? How to deploy docker-compose.yml file to a remote host?

2) How can I deploy new version of a microservice to a remote host? I am looking for a tool so that if I run

tool deploy app@2.33 [server1]

it would run the app image at version 2.33 on server1. Is there a tool for that?

For instance I was using a tool for meteor deployment meteor-up that sends the meteor app docker image to a remote host and runs it there - I am looking for the same thing but for a random docker image/docker compose.

I can’t find what is the best practice now in the docker ecosystem to deploy an image to a simple server without AWS/Beanstalk or any other cloud environment. Should I just ssh into it, copy docker-compose and run docker-compose up -d ?

Any thoughts appreciated, thank you!

Let's take your questions one-by-one:

  I want to be able to specify different versions of microservices for different prodction servers. How should I approach this?

There are many different ways to handle this.  One possibility be to "tag" your docker images and have different docker-compose files that use these images.

  Should I have one general docker-compose file or one file with specified versions for each production environment?

Depends.  How many environments do you have?  What I've often seen is a single docker-compose that references a separate environment file.

  Where should I keep docker-compose? In a repository?

Again this depends.  A standard enterprise best practice would be to store this in a repository so that it is naturally version controlled, and can easily be accessed by others.

  How it is copied then to a remote host? How to deploy docker-compose.yml file to a remote host?

For deploying a docker-compose.yml it depends on the target architecture.  Since you mentioned Linux you could use scp to copy the file.

  How can I deploy new version of a microservice to a remote host?

The industry leading tool for this right now is Kubernetes.  If you go this approach your docker-compose could be morphed into a Kubernetes deployment.  Using just docker-compose you will need to somehow detect that a new container is available and pull it.  Watchtower or a similar tool may be helpful.

Based on what you've described I would recommend taking a look at the interactive Kubernetes tutorials, and seeing if that is ultimately an easier approach than everything I outlined above.


Why is developing Ansible Playbooks using Roles a best practice? It's a best practice to develop Ansible Playbooks with reusable roles.

From what I've seen, people end up putting all their roles into a single complex files anyways.  So it doesn't really fix the problem there.  I've read they can be split into multiple files, but now there's more to manage and it's no better than just writing multiple playbooks.

Especially considering you can reuse playbooks and tie them together with an Ansible Tower Workflow, I'm not seeing the benefit of including an additional layer of abstraction and complexity for roles.

Can someone tell me if I'm crazy for thinking this?

EDIT:

I've had some more thoughts after reading Bruce's answer.

I don't want to be converting Ansible-Galaxy roles into playbooks, or make people I share with turn playbooks into roles.
Not everyone I share or collaborate with has access to Tower.
Testing

There are many reasons that using roles is better than using long playbooks, and very few reasons that using a single long playbook is better than using a role. These are almost always clear only at a certain scale. Team size, workload number and complexity and frequency of execution all come into play here and the larger any of them get, the more obvious it becomes that organising things hierarchically and modularly into roles is the right thing to do.

From the Ansible docs, there is an emphasis on re-usability:

  Roles are ways of automatically loading certain vars_files, tasks, and handlers based on a known file structure. Grouping content by roles also allows easy sharing of roles with other users.

This is also why Ansible Galaxy is potentially useful. One can summarise the effect of a long list of tasks, into a single concept - a role. Although the role may be considered as simply a long list of tasks -- and so not that different from a playbook -- the main benefit comes from the way it scopes variables. Out of the several analogies which may be applied, one may use the Object Oriented analogy. A role is like a class. The usage of that role, applied in a playbook, is like the instantiation of a class into an object. If you consider object-oriented programming useful, you will consider object-oriented infrastructure provisioning useful.

For me, however it comes down to testing. A playbook is tactical and is useful in specific situations. A good role is strategic and is written to solve general problems. Since good roles are modular, they should contain unit tests for various scenarios to prove to the downstream user that they do indeed solve the more general problem, as well as making it clear what is actually covered in the tests.

Typically roles should be maintained, with proper documentation:

what variables do they use ?
what scenarios are they written for?

and wide coverage:

several base operating systems
several deployment scenarios (docker, AMIs, bare-metal, etc)

and organise tasks cleanly:

use tags
dynamically import tasks based on execution conditons and user-provided variables

etc.

In short, I wouldn't consider it a fair comparison to compare a bunch of playbooks with the usage of different roles. When it comes time to refactor your playbooks into something maintainable, you will need roles.


How to Shutdown Docker Stack on Container Failure Say I have several docker services using swarm defined as such:

version : '3'
services:
    server:
        image : my_server:latest
        deploy:
            replicas: 5
            restart_policy: any
    database:
        image : my_db:latest
        deploy:
            replicas: 5
            restart_policy: any
    logs:
        image : my_logger:latest
        deploy:
            replicas: 1
            restart_policy: any

I then deploy this as a docker-stack as such. The services are server, database and logs. I make great use of the network abstractions provided by docker swarm, but my failure-case is a bit undockerish.

*In this case when all containers in database fail, then I want the entire stack to be shutdown or otherwise made unavailable*. Is there any built-in way to accomplish this task? Does the concept of a stack exist in a way for the individual services to access?

This question can be alternatively answered with a generalization : "How, with built-in docker features, can I be made aware of container failure"? Because I'm assuming there is no way to do exactly what I am asking.

If you don't want the containers to restart, you can set the condition: to none. See https://docs.docker.com/compose/compose-file/#restart_policy

On the swarm manager, you could use docker stack ps to check the status of the containers on a regular basis and shut down the stack if the database containers aren't running (using cron or whatever). For example:

docker stack ps --filter "desired-state=Running" your-stack |grep database || docker stack rm your-stack.

If containers are not able to restart, they shouldn't show up in the list once they fail, and once none are found, you can remove the stack.


TravisCI did not run after adding more configuration to .travis.yml file I added some AWS keys to my .travis.yml file so that it would deploy my application over to ElasticBeanstalk, but after I committed the changes to the file and pushed to master, nothing happened.

I thought perhaps I needed to restart build. This added my AWS keys, but no installation of deployment dependencies as I had expected and no deployment to AWS.

Not sure what happened here. Did I not wait long enough for TravisCI to start the new build?

This is my .travis.yml file:

sudo: required
services:
  - docker

before_install:
  - docker build -t danale/project-name -f Dockerfile.dev .

script:
  - docker run danale/project-name npm run test -- --coverage

  deploy:
    provider: elasticbeanstalk
    region: "us-east-1"
    app: "project-name"
    env: "ProjectName-env"
    bucket_name: "elasticbeanstalk-us-east-1-123456789"
    bucket_path: "project-name"
    on:
      branch: master
    access_key_id: $AWS_ACCESS_KEY
    secret_access_key:
      secure: "$AWS_SECRET_KEY"

I conducted a git add . && git commit -m "added travis deploy script" && git push origin master

No deployment afterwards. I don't see where I missed a step here:
https://docs.travis-ci.com/user/deployment/elasticbeanstalk/

One of the headaches of .yml files is the indentation. So the parse error I was getting in the Request tab was saying no newline added. I was not clear whether that meant it needed a new line or was missing a new line, but by running: wc -l .travis.yml I was able to detect that it was not outputting the correct number of lines.

I tried about five different ways to format the .yml file allowing it to keep me focused towards the end of the files line number where the error was pointing towards.

I happen to look up and realized the beginning of the deploy: script was not properly aligned, so I adjusted with a shift+tab and after I got it properly aligned, TravisCI recognized my changes and properly ran the script and successfully deployed the application to Elasticbeanstalk.


How to disable ping thread on jenkins agent One of the jenkins slave is going down very often. Found RCA for this issue.
It's because of the Ping thread

In the logs I fetch from agent, I have:

Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started at 1576130988405 hasn't completed by 1576131228405
    at hudson.remoting.PingThread.ping(PingThread.java:134)
    at hudson.remoting.PingThread.run(PingThread.java:90)

I wanted to disable the ping thread on the agent. In the documentation it's mentioned as -Dhudson.remoting.Launcher.pingIntervalSec=-1

But I do not understand how to execute this on remote machine.

I followed this link. but that solution is for JNPL. I am launching the agent on the node via ssh.

In the web UI, from the home page, navigate to Manage Jenkins > Manage Nodes > Configure.  This page has all of the health checks that Jenkins performs to determine if an executor is up.  The check you are concerned about is called "Response Time".  You can disable the "Response Time" health check on this page.

From the help text on the same page:

  This monitors the round trip network response time from the master to the agent, and if it goes above a threshold repeatedly, it marks the agent offline.

  This is useful for detecting unresponsive agents, or other network problems that clog the communication channel. More specifically, the master sends a no-op command to the agent, and checks the time it takes to get back the result of this no-op command.


Initial Configuration: Grafana and Prometheus inside docker and then AWS I'm a bit of a newbie with this, but I haven't been able to find an explanation. Here's my issue. I'm trying to get Prometheus and Grafana configured on an AWS Fargate instance. I'm testing inside a Docker instance on my desktop. Both environments were coming up, but Grafana was reporting no data.

My Datasource:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: direct
    orgId: 1
    url: http://localhost:9090
    isDefault: true

I started the container with this:

docker run -d \
    --name prometheus \
    --hostname prometheus \
    -p 8110:22 \
    -p 9091:9091 \
    prometheus:latest

When I couldn't figure out why Grafana reported no data, I restarted the container, adding -p 9090:9090 so that I could access Prometheus's graphing interface. And suddenly Grafana shows data.

Which has me deeply puzzled. I don't know how Grafana works under the hood, and so far I haven't found an explanation. So I'm wondering why port-mapping fixed the problem.

Do I need to configure Grafana to access Prometheus via an externally-resolved URL (that is, one that resolves in my browser, not from Grafana itself)?

I hadn't planned on exposing Prometheus itself to the world, but maybe I have to.

Additional Info: Within AWS, my Grafana charts all have a red warning indicator for each chart, and the rollover says "Not allowed to request resource". I haven't found any docs yet on what can cause this. Still digging.

Okay, it took time and help from someone else. I was going down the path of installing nginx, but that won't end up being necessary.

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy
    orgId: 1
    url: http://localhost:9090
    isDefault: true

This is the end change. originally I had access: direct. As I understand it, that means access to the metrics is via the browser. Switching it to proxy means that Grafana will look up the data. Which means the URL doesn't need to be externally-useful.

To a newbie, none of this was obvious, so maybe my two days of pain will help someone out in the future.


AWS: Application vs Classic Load Balancer My senario is this:

I am using 2 EC2 instances which I want reachable via urls as follows:

sub1.domain.com
sub2.domain.com

I am aware of the fact that AWS Application Load Balancers currently support host based routing so I guess this is now feasible.

Currently I am using 2 Classic Load Balancers on AWS to support access to those instances;

Is there a reason for me to switch to the host-based routing scenario of ALBs?

Perhaps the motive for this question is silly, but lately Amazon in its console, has grayed out the Classic ELB and marked it as

  Previous Generation

(I guess this wouldn't be a strong motive but it kind of discourages you a bit from using Classic ELBs henceforth...)

P.S. Haven't got the chance to test it, but I assume if I have issued an AWS wildcard certificate for *.domain.com, this would cover all target groups (i.e. hosts) under the ALB, right?

1) you are using 2 classic ELBs instead of 1 ALB
so costs savings can be a factor

2) using a layer7 loadbalancer vs a layer4 gives many features and capabilities not possible otherwise

you can see a comparrison here:
https://aws.amazon.com/elasticloadbalancing/details/#compare

3) yes that wildcard would fit all your needs (sub1/sub2.domain.com)


Multiple paths with different expiry time in gitlab-ci runners I'm having issues with parsing a build directory between stages using Gitlab-CI.

Gitlab-CI wipes the created build artifacts between stages which seems weird. I'm trying to store the build artifacts between the build and the test stage, however the build stage also has the build artifact which I want to keep and also the build artifacts which are required to run the next stage.

Is it possible to have multiple expiry times with different paths using the artifacts option?

I have tried the following, which only keeps the second definition of paths (the build/test* paths), and not the first paths (.dmg) declared.

  artifacts:
    paths:
    - build/*.dmg
    expire_in: 1 week
    paths:
    - build/test1
    - build/test2
    - build/test3
    expire_in: 15 mins

I have tried using the caches however can't seem to get that working... Any suggestions would be great appreciated!

This has been answered over on SO using a work around seeing as it doesn't seem possible according to the documents.

Basically, this can be done in 3 stages.

Stage 1: Build and store all artifacts.

build_stage:
  script:
    - build
  artifacts:
    paths:
    - build/*.dmg
    - build/test1
    - build/test2
    - build/test3
    expire_in: 15 mins

Stage 2.1: Do the next official stage of the job (ie run tests in my scenario) using the artifacts build/test1, build/test2 and build/test3.

test_stage:
  script:
    - test
  dependencies:
  - build

Stage 2.2: Running concurrently with Stage 2.1 you can just have an empty job but sets a new artifact expiry date.

overwrite_artifact_stage:
  script:
    - echo 'saving artifact'
  artifacts:
    paths:
    - build/*.dmg
    expire_in: 1 week


List encrypted RDS snapshots using CLI I have automated the RDS snapshot copy across the region, but the Snapshot which is encrypted gets fail to copy.
I have figured a way to copy an encrypted snapshot using

$  aws rds copy-db-snapshot   --source-db-snapshot-identifier arn:aws:rds:eu-west-1:XXXX:snapshot:XXX-2017-01-31-04-30      --target-db-snapshot-identifier mydbsnapshotcopy   --kms-key-id XXX --region eu-central-1 --source-region eu-west-1

I want to filter/test the snapshots if they are encrypted or not, if they are encrypted I will perform the above-defined operation.
I tried this which is not working.

$ aws rds describe-db-snapshots  --filter "Name=encrypted,Values=true"

How do I list/filter all the encrypted snapshots and move it to a file?

Either use the --query attribute

aws rds describe-db-snapshots --query "DBSnapshots[?Encrypted].DBSnapshotIdentifier"

If the [?Encrypted] does not work, try [?Encrypted == 'true'] as the first is from JMESPath improved filters. The quotes might need to be backticks in some case. It all depends.

Second way:

aws rds describe-db-snapshots --output json | jq '.DBSnapshots | map(select(has("Encrypted"))) | .[].DBSnapshotIndetifier'

One of them should work. Might need a bit messing around as I don't have your output to tune the jq command. Feel free to edit and fix the answer.


Running unmanaged Hardware Security Modules (HSMs) in the Cloud I have to admit to never having asked, or been asked, the question if it is possible to have a Hardware Security Module in a public cloud, by which I mean Google, Amazon or Azure.  Has anyone found any techniques for enabling organizations to use HSMs that they fully manage?

It seems to me that the two concepts, Cloud and HSMs, are fundamentally at odds with each other - because cloud generally involves "outsourcing" or transferring the risk of operating hardware to the cloud service provider.

There is clearly a middle ground in terms of fully managed HSMs as you find in Azure and AWS:

Azure KeyVault: Use Key Vault and you don’t need to provision, configure, patch and maintain HSMs and key management software. Provision new vaults and keys (or import keys from your own HSMs) in minutes and centrally manage keys, secrets and policies.
AWS CloudHSM: AWS CloudHSM is a cloud-based hardware security module (HSM) that enables you to easily generate and use your own encryption keys on the AWS Cloud. With CloudHSM, you can manage your own encryption keys using FIPS 140-2 Level 3 validated HSMs.

In addition there are some non-HSM based solutions to key management:

Cloud Key Management (Google): Cloud KMS is a cloud-hosted key management service that lets you manage encryption for your cloud services the same way you do on-premises. You can generate, use, rotate and destroy AES256 encryption keys. Cloud KMS is integrated with IAM and Cloud Audit Logging so that you can manage permissions on individual keys, and monitor how these are used. Use Cloud KMS to protect secrets and other sensitive data which you need to store in Google Cloud Platform.
Various Security Appliances available in all of the cloud marketplaces.

Has anyone found any techniques for enabling organizations to use HSMs that they fully manage?

So having gone backwards and forward over this for a couple of weeks, Azure has confirmed to me in-person that the only way to utilise FIPS-140 Level 2 certified hardware security modules in Microsoft Azure is to use Azure Key Vault.


API gateway and service discovery on Amazon AWS I'm trying to build an application infrastructure on Amazon AWS following a microservice pattern.

However the configuration API gateway and service discovery tool together completely puzzles me.

According to a pattern I need both (and I understand that). I decided to go with Amazon API gateway as API gateway and Hashicorp Consul as service discovery.

From what I see in the configuration guide on Amazon, API gateway configuration is pretty much straightforward: I specify path and URL to follow and that's it.

How in the world do I explain API gateway to get information from Consul?

PS. There is a Consul quickstart guide, but it is just a deployment instruction. There is nothing about connecting it to the API gateway.

It turns out there is no existing solution for such a configuration.

I have to develop something on my own.

A nice source for help might be a series of articles on the Amazon developer blog and REST API documentation for the Amazon API gateway service.


Jenkins - Active Choice DynamicReferenceParameter not returning list I have been trying unsuccessfully to get DynamicReferenceParameter of the Active Choice plugin to show previously selected values as list, however it either shows them on one line or it treats each symbol as a new element in the list.
hosts_script = '''
return['host1', 'host2', 'host3']
'''

properties([
    parameters([
      [$class: 'ChoiceParameter',
        choiceType: 'PT_MULTI_SELECT',
        description: 'Select one or more hosts',
        filterLength: 1,
        filterable: true,
        name: 'hosts',
        randomName: 'choice-parameter-5631314439613978',
        script: [
            $class: 'GroovyScript',
            fallbackScript: [
                classpath: [],
                sandbox: true,
                script:
                    'return[\'Could not get hosts\']'
            ],
            script: [
                classpath: [],
                sandbox: true,
                script: hosts_script,
            ]
        ]
    ],
    [$class: 'DynamicReferenceParameter',
        choiceType: 'ET_UNORDERED_LIST',
        description: 'These are the selected hosts',
        name: 'Selected hosts',
        omitValueField: false,
        randomName: 'choice-parameter-5693388460832175',
        referencedParameters: 'hosts',
        script: [
            $class: 'GroovyScript',
            fallbackScript: [
                classpath: [],
                sandbox: true,
                script:
                    'return[\'Could not get hosts\']'
            ],
            script: [
                classpath: [],
                sandbox: true,
                script:
                    """
                    //def list = hosts.replaceAll( '"','\\'' )
                    //List list = hosts.collect{ '\\'' + it + '\\','}
                    //def list1 = new ArrayList(Arrays.asList(hosts))
                    List list = hosts.collect{ '\\'' + it + '\\','}
                    //String list = hosts.inspect()

                    return list
                    """
              ]
          ]
       ]
    ])
 ])

I can get this  • host1,host2
Or this:
 •'h',
 •'o',
 •'s',
 •'t',
 •'2',

Where I want this:
 • host1
 • host2

At the very end you will see what I have been playing with ... the biggest problem for me is that is impossible to use println to troubleshoot.

I figured out what was going on. The return from hosts var was a single string with comma separated hostnames host1, host2, I had to convert it to List  using the comma as delimiter. The following post How to convert comma-separated String to List? helped me with that task.
Basically the last section now reads
                script:
                """
                List<String> list = Arrays.asList(hosts.split("\\\\s*,\\\\s*"));
                return list
                """


How to manage 10+ team environments with Helm and Kubernetes? I'm looking for feedback on how we can improve our team environment and staging deployment process.
We currently have 10 agile teams, and set up our Kubernetes deployments through Helm as follows:

All our helm charts are in a single git repo. At the top layer of the repo we have a collection of bash scripts (deploy.sh, buildSecrets.sh, and buildConfigMaps.sh) and some json configuration files. (namespaceCharts.json, secrets.json, configmaps.json)

The two main configuration files are a long list of secrets, which are encrypted, and a long list of config map environment variables, 80% of which are shared between helm charts. The namespaceCharts.json defines which charts need to be deployed into each namespace grouping.

For each environment that we deploy into, we clone the repo, change the values of the  secrets.json and configmaps.json and then run the scripts passing in our base namespace name. (For example, the team name)

This results in us basically creating a series of namespaces such as teamA-frontend and TeamA-backend each of which contains the right microservices with the connection strings to the correct DB etc.

We are currently hitting a few main roadblocks.
 1. It's hard to deploy to multiple clusters or new clusters.
 2. It's hard to get a single source of truth for the configurations.
 3. It's hard to set up a brand new Team environment because we have to go through the entire configuration file to see what needs to be changed or kept.

I have come up with three ways of moving forward, all of which "feel wrong" to me.
1.  Store all the environment configurations in separate "configurations" git repo.
2. Create a configuration service with a DB table defining each environment, and create a pub/sub feature to dynamically grab configuration at runtime, as well as dynamically grab configurations in the .sh scripts.
3. Use ansible or something similar to store the scripts and configurations for each environment.

Are we doing this all wrong? Are we missing some obvious best practice to do this better? Are one of those three ideas actually the right approach?

My 2 cents.  We have similar requirement like you do.  We end up having multiple values files in the form of values-<namespace>.yaml and it should come with the defaults set to values.yaml

We also have some shell scripts to deploy to the namespace.  Something like this deploy.sh --namespace app.

It is not without its problem, the number of files are steadily growing.  But so far we are ok with it.


How to read array and loop server selection and deployment I have configured jenkins parameter deployment job using active choices parameterize plugin. In my dev server section have two servers called, 192.168.94.139, 192.168.94.140

I want to read this array input and deploy to both servers. As an example if I select both servers artifacts should be deployed to both servers. How can I read those inputs to fulfill my requirement. Until this point I wrote an array to read inputs but it prints like this

  192.168.94.139,192.168.94.140

my_array=($SERVER_SELECTION)
for i in ${my_array[@]}
do

   echo $my_array

done

Look this:

#!/bin/bash
#Get servers list
set -f
string=$DEPLOY_SERVER
array=(${string//,/ })
for i in "${!array[@]}"do
      echo "Deploy project on server ${array[i]}"
done

I found this link


Given a blank slate, tool recommendations? so I have 8 months of DevOps experience previously working as a desktop engineer for two and a half years.Starting a new role in the education sector, recently they brought their IT department which previously worked in different projects/faculties and now they have all been brought together.So this is where I come in to help dictate the application development process by designing and maintaining the infrastructure to build an efficient route to live regardless of the project. So this is essentially a blank slate.

As a very high level overview

So my ideal setup would be creating a production environment for whatever project, replicating the stack and using that as UAT. This would be done through Terraform and Ansible. The Dev estate I am not too sure how this is going to be setup yet but I was thinking about learning docker and k8s to bring up a dev estate.

The way I would look into setting up a CI/CD pipeline using Jenknnis and actioned through Azure DevOps as this is what they use, I would ensure code pushed from the DEV branch is deployed to UAT and tested then deployed to replicate production servers and once tested DNS is switched over to the new servers and the old ones are brought down.

The way how the code is deployed is through building the codebase on the same server Jenkins would be setup on, sending a tar'd file of the code to AWS CodeDeploy to be taken to the next environment.

Through caveats include no changes or access will be allowed on production or UAT servers, any requested changes will be done through terraform and ansible to keep things consistent.

Personally with my limited experience I have used Terraform and ansible and I have felt this provides alot of the work to have that infrastructure as code system in place. At the same time I understand there are tools like chef, puppet and salt which would probably make the job easier but that is where my question is what would you guys want and how would you want to implement it. I should finally mention we seem to be using Azure DevOps as the main tool to setup and manage the pipelines.

This question runs two risks:

of being marked as "primarily opinion-based", and
its answers might age poorly.

However, I think it's useful to express an opinion, if only for posterity; if and when things change, there may be better answers -- but there will always be blank slates.

I would split the problem up into four sections:

Infrastructure
Application Building, testing and integration
Deployment
Monitoring, logging and analytics

Furthermore, let's make this cloud-independent (so, this toolkit should be valid for any or all of AWS, GCE, Azure etc, as well as on-prem OpenStack, bare metal cloud (e.g. Packet, etc).

Infrastructure

Infrastructure components consist of

Your environments (pre, prod, integration, testing, etc - however you define them). These should be built with Terraform modules.

Each module in its own repo
Consider carefully separation of state, so that you can easily create and destroy components and keep costs down

"DevOps" infrastructure. This includes:

code repos
CI/CD instances (controller, agent pools, etc) - e.g. Jenkins master with cloud agent pools.
Deployment runners (see later).
Monitoring, logging and analytics.
Again, build these with terraform modules. As a example, have a Terraform module for each type of application or component.

Team, messaging and support infrastructure. This is often overlooked, but choosing the right tool for the job on Day 2 is hard. It's better to communicate to the stakeholders how issues and change requests should be made. I would venture a combination of GitHub issues and Slack at the very least.
secrets management. You will need to pass around a lot of secrets, if you want to foster collaboration. In order to do this securely, you will need a place to manage them and delegate access to and usage of them.

Building, testing, integration.

These happen in Jenkins. Each application should contain a subdirectory .pipeline with the parts necessary to build it. The part which concerns us here is everything before prod.

I would suggest a pipeline as code approach with a Jenkinsfile in the repo declaring the steps for

building the application:
Testing the application

Unit tests
Functional tests
Integration tests
(Regression tests)

Building artifacts

building and signing artifacts
Shipping them to the artifactory, with new version info and other metadata

Building instance images (if your application is "installed" in an environment, and not serverless).
Enforcing changes to the Infrastructure (terraform plan/apply)
Enforcing changes to the deployment configuration (capacity, etc)

At the end of this stage, you'll have a properly configured environment (say, pre or prod), an artifact, and an image which you can deploy. If need be, you can run performance tests (with Taurus) or acceptance tests (with cucumber) on them with a job in Jenkins.

Deployment

I would vote for a hard separation between the previous phase and this one. Deployment should be done with a different tool and the right tool for the job is in my opinion Spinnaker. The configuration for the deployment should be kept in the same repo as the application (so that developers can serve themselves). Deployments should be done on every change to the codebase, declaratively -- Spinnaker has a JSON syntax for this, making it easy to check consistency and correctness of the deployment scenario.

Monitoring, logging and analytics

As mentioned before, Day 1 is a blank slate, but it is quickly dirtied. The DevOps doesn't stop on Day 2 (ie, when apps are deployed), and it would be wise to plan ahead.

The SRE book gives good advice for instrumenting and monitoring the applications you will be deploying, but this space is very crowded with services and tools claiming solve your monitoring and alerting needs. Again going with the principle that we should be able to deploy anywhere, I would go with a Prometheus for indicator and metric extraction and Grafana for displaying setup. However, you should definitely consider this part wisely.

So - the toolkit looks like:

Vault for secrets
GitHub for code, projects, tasks and issues
Slack for communication
Terraform for infrastructure
Ansible for configuration management
Packer for image baking
Quay for (docker) image registry
Inspec for compliance and correctness of state
Cucumber for Acceptance Testing
Jenkins for pipeline execution
Spinnaker for Deployment
Prometheus for metrics Grafana for visualisation and dashboards
Taurus (Blazetool) for performance testing

Is this a lot of tools? Yes.

Will you be able to deploy anywhere? Also yes.

Will you need all of them? probably not. Also, I've chosen tools which are mostly free to use and open source.

Also note: in this scenario, the entire state of all applications can and should be represented as code, and can thus have semantic versioning (or other kind of versioning which makes sense to you). Any changes to that state can be represented by changes to the code, and can thus have good source code management principles applied to it, such as code review, automated testing, etc.


Cross Account EC2 Access from within Amazon Workspace I have a requirement to setup a bastion host (Amazon Workspace) to connect to an ec2 instance living on a different account.

As Amazon Workspaces cannot assume a role the way EC2s can, A cross account policy cannot be used to setup a link between them.

Both Amazon Workspace and EC2 instance live in their respectve VPC's in separate accounts. Is there a Cross account policy like solution which I can use to link these two?

P.S. I cannot create a Workspace in the same VPC/Account because the directory service is only available in one account but the EC2 which needs to be connected to is in other. Also, the EC2 cannot go public as well.

Use a VPC peering between the two accounts so that your instance from account A will be able to communicate with account B.
This will work if you have different subnet CIDR blocks in the two accounts. If you have the same (i.e. in both VPC, your subnets are in 192.168.0.0/16), then change the one that has the least impact and proceed with a VPC peering.
Finally, you will have to configure your security group to allow the bastion to communicate with the other instance.
Here are more details for AWS VPC peering: VPC peering


CIS Benchmark for Docker and Docker Bench for Security: why, how and when? I just learned about Docker Bench for Security and CIS Benchmark for Docker Community Edition.

Why, how and when am I supposed to use these tools?

Larger organizations are concerned that containers introduce more security risks than traditional VMs as there isn't any hypervisor segregation. Developers running arbitrary images, that run as root, that they don't bother to security patch, looks worse for security team's than automatically patched VMs controlled by ops teams without containers.

Misconfiguration of docker demon itself can also introduce security risks. It's fairly common for complex software to have security tuning tools and to run them as part of ”operational readiness checks” of the live environment before going live as a security audit. Then also run them on some periodic cycle or after any major upgrades of the software that is being audited. The tools you name will be very useful for that purpose.

Software vendors who build container orchestrators are actively working in removing any dependency on needing to run docker demon see cri-o.io. Security, footprint and fork time are areas that vendors are competing on. CoreOS was a big innovator in this space before they were acquired you can look at some of their architecture presentations to get a flavor of what containers at scale workout docker looks like.

From a devops perspective we should empower devs to run security tools all the time to fix issues early. In practice devs might be using Docker for Windows locally but the live env would be a container orchestrator possibly running rkt. In reality devs in large organizations are not going to security scan infrastructure they didn't create and dont have root on. Even at a startup not every dev can be familiar with all of the tooling and software running in production. An external security audit is a smart investment.

IMHO if your teaching containers from a developer perspective it's worth pointing out such tools. Certainly devs in my experience are woefully unaware that running containers they find on docker hub as root isn't the smartest way to be secure. I would only expect to see such tools to be covered in depth by courses teaching security analysts how to audit systems that use Docker.


Azure Devops: Filter Test Projects in Nuget Build Pipeline I'm working on a nuget build pipeline.  My problem is that my integration test project is being packaged and deployed to the azure artifacts, while my unit test projects are not.

The integration test project is a console application.  I want it in the project, but I don't want it to deploy into the artifacts.

My Projects in the solution:

MyProject.Common
MyProject.Common.UnitTests
MyProject
MyProject.UnitTests
MyProject.IntegrationTests

I am using .net core v2 steps

Restore
Build
Test
Publish
Pack
Push

Test Path to project (works)

  **/*[Uu]nit[Tt]ests.csproj

Publish Path to projects (works)

  !**/*[Tt]ests.csproj
  **/*.csproj

Pack path to project (does not work - filters unit test projects, but not integration test)

  !**/*[Tt]ests.csproj;**/*.csproj

I noticed that the description for the filepath properties of the pack command is different than the descriptions in the test and publish commands:

Pattern to search for csproj or nuspec files to pack.

You can separate multiple patterns with a semicolon, and you can
make a pattern negative by prefixing it with '-:'. Example:
**/*.csproj;-:**/*.Tests.csproj

So perhaps try -:**/*[Tt]ests.csproj;**/*.csproj instead. I don't think this explains why the unit tests are excluded and the integrated tests are not, though.


Robocopy is Running Slowly on Jenkins Server I'm working to automate the deployment process for a Fullstack application using Jenkins. The app is deployed on a Windows Server. The .cmd script I have to automatically deploy the app is this:

@ECHO OFF
REM SET THE PARAMETERS FOR THE SCRIPT TO RUN
SET CONFIG_ENV=%1
SET SERVER_DOMAIN=%2
SET ADMIN_LOGIN=%3
SET ADMIN_PASS=%4
SET ARTIFACT_DIR=%~5
SET DROPFOLDER_PATH=%~6
SET DEPLOY_PATH=%~7
SET SERVICE_NAME=%8

REM ECHO THE PARAMETERS OUT TO THE CONSOLE (SO WE CAN SEE WHAT'S GOING ON)
ECHO ...
ECHO RUNNING WCF AUTO DEPLOYMENT. PARAMETERS ARE:
ECHO CONFIG_ENV      : %CONFIG_ENV%
ECHO SERVER_DOMAIN   : %SERVER_DOMAIN%
ECHO ADMIN_LOGIN     : %ADMIN_LOGIN%
ECHO ADMIN_PASS      : %ADMIN_PASS%
ECHO ARTIFACT_DIR    : %ARTIFACT_DIR%
ECHO DROPFOLDER_PATH : %DROPFOLDER_PATH%
ECHO DEPLOY_PATH     : %DEPLOY_PATH%
ECHO SERVICE_NAME    : %SERVICE_NAME%
ECHO ...

REM SELECT THE LATEST SET OF RELEASE ARTIFACTS FROM THE DSL
FOR /F "delims=" %%i IN ('dir "%ARTIFACT_DIR%" /b /ad-h /t:c /od') DO SET latest_build=%%i
ECHO DEPLOYING ARTIFACTS IN: %ARTIFACT_DIR%\%latest_build% (latest build)
ECHO DEPLOYING TO "\\%SERVER_DOMAIN%\%DROPFOLDER_PATH%"
ECHO ...
REM LOGIN TO DEPLOY TO THE SERVER
net use \\%SERVER_DOMAIN% %ADMIN_PASS% /USER:%ADMIN_LOGIN%

REM COPY ALL THE RELEASE ARTIFACTS TO THE DROP LOCATION ON THE SERVER
ECHO COPYING ARTIFACTS TO DROP FOLDER
ROBOCOPY "%ARTIFACT_DIR%\%latest_build%" "\\%SERVER_DOMAIN%\%DROPFOLDER_PATH%" /MIR /is /it /np /r:1 /w:1 /eta /tee /copy:DT
ECHO ...

REM STOP THE SERVICE SO THAT WE CAN UPDATE THE FILES
ECHO STOPPING THE SERVICE
SC \\%SERVER_DOMAIN% STOP %SERVICE_NAME%
ECHO ...

REM COPY ALL THE .DLLs and .EXEs FILES
ECHO COPYING .DLL AND .EXE FILES
ROBOCOPY "\\%SERVER_DOMAIN%\%DROPFOLDER_PATH%" "\\%SERVER_DOMAIN%\%DEPLOY_PATH%" *.dll *.exe /is /it /np /r:1 /w:1 /eta /tee /copy:DT
ECHO ...

REM COPY THE APPROPRIATE CONFIGURATION FILE FOR THE ENVIRONMENT
ECHO COPYING CONFIGURATION FILE
ROBOCOPY "\\%SERVER_DOMAIN%\%DROPFOLDER_PATH%\Configuration_Files\%CONFIG_ENV%" "\\%SERVER_DOMAIN%\%DEPLOY_PATH%" *.config /is /it /np /r:1 /w:1 /eta /tee /copy:DT
ECHO ...

REM START THE SERVICE BACK UP AGAIN
ECHO STARTING THE SERVICE
SC \\%SERVER_DOMAIN% START %SERVICE_NAME%
ECHO ...

REM AND WE ARE DONE
ECHO SCRIPT COMPLETE

If I run the script directly from where it's located on the Jenkins Server, the script finishes in a matter of seconds. However, when I queue the script to run from a Jenkins job, the ROBOCOPY command takes roughly a minute to copy each file.

If I just run the same script directly on the (exact same) Jenkins server, the process runs a lot faster.

What is causing the process to run slowly and how can I make the script run faster when executed from the Jenkins Job?

This issue is resolved.  Robocopy has a lot of smart features and functions that allow it to check and see if the files being copied are actually different.  Unfortunately that comparison is where the time is really going.  Once we changed the script to delete the files from the destination folder first (every time) - we should have a repeatable process that runs in ~10-15 seconds.


Docker convenience script stuck after "Scanning linux images" I have a Raspberry Pi 2B and a fresh install of Raspbian.

I am trying to install Docker, and following the instructions on the site (https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script), I used the convenience script just like it's written there.

This is all the output I get:

# Executing docker install script, commit: 26ff363bcf3b3f5a00498ac43694bf1c7d9ce16c
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c curl -fsSL "https://download.docker.com/linux/raspbian/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key output should not be parsed (stdout is not a terminal)
+ sh -c echo "deb [arch=armhf] https://download.docker.com/linux/raspbian buster stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ [ -n  ]
+ sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
Scanning processes...
Scanning linux images...

I let it run for more than a half hour and nothing changes.

How do I resolve this?

Just found this question from 5 days ago because I ran into the same thing on a RPi 4, also using the convenience script.

Just waiting a bit did not seem to help. So I killed the process after a while. And Docker is running just fine, now.


Should I give an assessment to DevOps Candidate We're a software team currently trying to hire a DevOps Engineer.

We have an assessment for Software Developer candidates and we're currently evaluating the idea of having an assessment for our DevOps Engineer candidates. Since we have not hired DevOps people before and barely know a few, it would help us greatly.

What would be your advice ?
If you support there should be an assessment, then what would be the content ?

EDIT(Additional Info):

We need to hire a DevOps Engineer to bootstrap our all DevOps process/culture.
We have more than 100 servers. Around 20 different apps on them.
We want him/her to;

Install a main devops tool like Jenkins or Teamcity
Build the pipelines for all this apps and servers
CIntegration, CDelivery, CDeployment depending on what app does
help migrate to Linux and/or docker

About assessment

It must be about building and deploying a few apps interacting with each other to different targets with different configurations.

EDIT 2(answers):

Thanks for answers. While reading them, I'll answer the questions to make the backstory clearer.

@taleodor

Developers including me are doing provisioning/build/deployments. We're very unhappy about that.
Unfortunately, we can only hire one person for all our expectations.

We have plenty of time to put everything together, so it should not be problem for anyone who's willing to do all the stuff.

Assignment will be take-at-home and candidates will be given a few days to complete. No problem about that. After all assessment is complementary to interview.
I know a few DevOps people. I may get their help for evaluating assessments.

@030

I agree the devs automating their stuff is much better than another person doing that. But we have a big backlog about devops. We need someone to show us the way while doing the bulk of it himself/herself. We are not considering to totally forget the process. We want exact opposite: we want to do things right but we don't know where to start and don't have time.

I installed Jenkins. Set up pipelines for some of the projects. But I am a senior developer. Neither me nor my executive wants my all time spent on this.

EDIT 3(What I came up with):

This is roughly the assignment I brought together. What do you think ? What else can be here ?

Install Jenkins on a platform on your choice
Dockerize .net project given.
Build two pipelines with unit test automation.
Build and host database.
You must be delivering us your scripts and an account to jenkins.

Yes, there absolutely should be an assessment, however it would be a good idea to agree upfront with one of the few DevOps people you know to validate the content and results.

Regarding the contents of the assessment, there are some questions still remaining after your edits. Key one is how your operations are done. Possible options are: you already have someone doing your server infrastructure and operations and looking for a purely CI/CD person, or you do not have good operations support (i.e. developers doing that and you're unhappy) and you need DevOps person to do server management too. Note, that if it's the later case, you'd probably need more than person over time but you need somebody senior to start with.

Note, that for larger organizations you'd find more specialization within DevOps (i.e. things like Release Management, DevSecOps and others), but for smaller companies those two - CI/CD vs Infra Operations are two main ones in my experience. You still need to keep those differences in mind if you're considering taking someone from a larger company.

Now, after answering those questions, you need to first establish the seniority level of a person you're looking to hire. If it's your first DevOps person - you would probably want somebody more senior, but it depends on other people in your existing team. Next you scope your assessment to the on-job tasks. My take - always give take-at-home assignments, I usually give it to candidates over weekend and I make sure they take no more than 3-4 hours.

For CI/CD case you may ask them for a simple CI pipeline in their tool of choice, built from scratch over some sample project. You should add extra questions about failure handling, parallelizing, caching, handling of automation tests.

For CI/CD + operations case you should add tasks about shell scripting, cloud security, terraform.

Also for the first DevOps / senior person it's always a good idea to add some questions about interacting with other teams. I.e. what developers would need to do and how the interaction process with them should be built.

Again, start with one of the few DevOps people you know and get them onboard as a consultant to assess and hire that first person, otherwise you're taking a lot of chances there - and DevOps is a pretty sensitive part of your business.


Jenkins upgrading and plugins ordering I have Jenkins 2.89 installed with many outdated plugins as well. What is the best path for upgrading both the plugins and Jenkins itself? Should one proceed the other?

I am leaning toward upgrading the plugins first and in batches to limit the risk. The system is being backed up in its entirety in case things go sideways. I would like to avoid that if at all possible though.

Update: I was able to upgrade to 2.138 without it breaking anything.

I can only speak from personal experience rather than best practices, but in our company we'll make a duplicate jenkins instance, then upgrade that. We do main upgrade first, then plugins second, then run testing to make sure our core jobs still work. If all is good, we'll then upgrade the real jenkins during off-hours.

We have 90+ jobs so testing them all is unrealistic - we made an effort to identify 8-10 jobs that span 90% of our use cases and run those on the standup jenkins to affirm everything is healthy first.

I'd highly advise doing something similar if you're using your jenkins in production scenarios. You never know what might break - we had one upgrade where a bug spun up 150+ aws ec2 instances in ~10mins instead of the 3 jenkins actually needed... another time all our unit and e2e testing of our code base broke because it turned out one of the output parser plugins had upgraded to become incompatible with our output format. We've even had our Bitbucket integration go sour and stop kicking off jobs on every push.

Sometimes upgrading all plugins at once can cause confusion - we had one scenario where something broke and finding the cause was difficult because it could have come from one of two different plugins. We ended up spinning up two additional jenkins and upgrading either plugin to identify which was the cause. In general though, upgrading everything at once will pretty quickly show its flaws with a little testing.

No matter what you decide, take a backup first!


How to get kubernetes pod network CIDR I have a running k8s cluster initialized with kubeadm.

On initialization, I did not pass the option --pod-network-cidr

How do I get the CIDR of the pod network

I tried

Looking at the /etc/kubernetes/manifests/kube-apiserver.yaml which seems to be the manifest for kube-apiserver used by kubeadm but that information is not included
Looking at the kubeadm document but I couldn't find a default value

This command will give you the pod CIDR addresses for each of the nodes in your cluster.

kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'


Hashicorp vault not writing out KV I am trying to set up a Hashicorp Vault server and have ran into some more road blocks. I can not get it so that I can write/read data either locally or through a domain name (I know the risks of have it available to the web and for this/testing it is fine).

Error message:

Error writing data to secret/keys: Put http://localhost/v1/secret/keys: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

Here is my config files:

/etc/vault.hcl

backend "file" {
        path = "/var/lib/vault"
        address = "127.0.0.1:8200"
        scheme = "http"
        redirect_addr = "http://mydomain.com:8200"
        VAULT_ADDR = "http://mydomain.com:8200"
}

listener "tcp" {
        address = "http://mydomain.com"
        tls_disable = 0
        tls_cert_file = "/etc/letsencrypt/live/mydomain/fullchain.pem"
        tls_key_file = "/etc/letsencrypt/live/mydomain/privkey.pem"

}

/etc/systemd/system/vault.service

[Unit]
Description=a tool for managing secrets
Documentation=https://vaultproject.io/docs/
After=network.target
ConditionFileNotEmpty=/etc/vault.hcl

[Service]
User=vault
Group=vault
ExecStart=/usr/local/bin/vault server -config=/etc/vault.hcl
ExecReload=/usr/local/bin/kill --signal HUP $MAINPID
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
SecureBits=keep-caps
NoNewPrivileges=yes
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

Some other configuration:

VAULT_ADDR=http://mydomain.com

Some additional info:

I used the command sudo vault operator unseal -tls-skip-verify because I could not get it to work over HTTPS

Command I used to configure the server vault server -config=/etc/vault.hcl

You have your VAULT_ADDR set to HTTP, but vault is listening on TLS - the error you are receiving is the HTTP request receiving a TLS handshake.

Try changing VAULT_ADDR to https://mydomain.com.


Install minimal dependencies with apt I have a simple Dockerfile which looks like the following

FROM debian:stretch-slim as builder
COPY . my_project/
RUN apt-get update && apt-get install -y build-essential libpqxx-dev libtbb-dev

Running this with docker build -t test_proj . I was dismayed to see that apt is trying to install a huge amount (583MB unpacked) of extra libs, including things like python and sqlite (which sound completely unnecessary for pqxx and tbb). I ran apt-rdepends over each and sure enough neither python nor sqlite were in there, so there are surely more 'spurious' libs being installed by apt

Why is this happening and how can I avoid this? I plan to use a multistage  to ditch the build tools by the final image anyway, but this is slowing down builds so I'd like to fix it

Have you heard about the distroless?
See this link too.


What is the purpose of AssumeRolePolicyDocument in IAM? I'm confused as to what the purpose of AssumeRolePolicyDocument is.
It seems like to create a role all you need is a parent name that you can attach policies to.
It seems like AssumeRolePolicyDocument is that parent name but somehow there's more to it.
What is this doing?
"AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": ["lambda.amazonaws.com"]
            },
            "Action": ["sts:AssumeRole"]
        }
    ]
}

The purpose of the AssumeRolePolicyDocument is to contain the trust relationship policy that grants an entity permission to assume the role.
In your example it's granting the Lambda service the ability to assume
References
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html
https://docs.aws.amazon.com/cli/latest/reference/iam/create-role.html


Customizing deploy key and ssh wrapper path for git cloning through application_git cookbook I am using Chef 12.0.3 for managing servers. Earlier I was using application cookbook for cloning repository. But now in higher version of application cookbook, this support is moved to new cookbook application_git.

Issue:

I want to change the path of deploy key
Contents of ssh_wrapper
Path of ssh_wrapper.

What I have tried (Current code):

application 'accounts' do
  action :deploy
  path <somepath>
  owner 'nobody'
  group 'nogroup'
  git 'name' do
    repository <repopath>
    revision <value>
    deploy_key deploy_key
 end
end

I am not able to find if I need to pass path using ssh_wrapper_path or ssh_wrapper or deploy_key_path. All are failing.

Currently, I am able to get keys and wrapper but at a wrong place.

Quoting from the cookbook README:

  deploy_key – SSH key to use with git. Can be specified either as a
  path to key file already created or as a string value containing the
  key directly.

And:

  The application_git resource deploys code from git. It extends the
  core git resource to support deploy keys and disabling strict host key
  verification.

So checking the core git resource documentation:

  ssh_wrapper Ruby Type: String

  The path to the wrapper script used when running SSH with git. The
  GIT_SSH environment variable is set to this.

so something like this (untested) should do (using the wrapper in the cookbook) :

cookbook_file '/path/to/my/desired/ssh_wrapper.sh' do
  source 'my_ssh_wrapper.sh'
end

application 'accounts' do
  action :deploy
  path <somepath>
  owner 'nobody'
  group 'nogroup'
  git 'name' do
    repository <repopath>
    revision <value>
    ssh_wrapper '/path/to/my/desired/ssh_wrapper.sh'
    deploy_key deploy_key
 end
end

You can of course do the same with the deploy_key, but using something like chef_vault or an encrypted databag sounds better to avoid commiting deploy keys within the cookbook.


Persisting data in docker context I am new to docker and flask web framework, my docker context has a flask file, a mod_wsgi file, and the python script. As I am new to flask, there are a lot of small changes which I keep doing. For any change to the docker context to take into effect I have to:

Kill the container and prune it out of the system.
Make changes to the Dockerfile and then rebuild the container.
In some cases I have to even remove the image and rebuild it as the changes are not taken into effect if cache is retained.

Is there any way in which I can have the changes into the docker context taken into effect during run time. I tried to have the docker context shared as a volume as that is the preferred way to persist data in Docker containers and services, but that did not work, any changes still were not reflected.

Could you please let me know of any way I can achieve this. Any suggestion would be greatly appreciated.

Thanks and regards,
Shailesh

Mounted volumes can be used to preserve state between runs, apart from that Docker is stateless by design. To make a local directory available inside the Docker container, you can use the -v option, for example:

docker run -v /path/to/local/dir/:/mountpath/inside/docker ...

But if I understand your question correctly, you are asking for a faster feedback cycle while working on the Dockerfile. For that, I would recommend to start the container, and attach an interactive shell to try out your changes. Once you made some progress, update the Dockerfile.

$ docker ps
CONTAINER ID  IMAGE        COMMAND  CREATED           STATUS        PORTS  NAMES
f1217635695d  ubuntu:18.04 "bash"   23 seconds ago   Up 21 seconds         stoic_austin

$ docker exec -it f1217635695d bash
root@f1217635695d:/# ...

Also note that proper use of caching can speed up rebuilds of the Docker container after changes significantly.

  (from your question)
  1. Kill the container and prune it out of the system.

Pruning should not be necessary unless you run out of disk space. It should be sufficient to update the tag, for example:

docker build -t test .
docker run --rm -it test

test:latest will point to the last image id that was created. docker images will show that the previous images are not deleted, but as said, if you have enough disk space available, it is not a big concern.


Continuous deployment of package to virtualenv My colleagues don't want to create their own virtualenv and deploy my tools from git on their own as part of their automation.  Instead they want the tools pre installed on a shared server.

So I was thinking of making a /opt directory, putting the virtualenv in there and then pulling from git every hour to force an update of the python package.  I am not version tagging the tools currently and instead would just ask pip to force the install every time.

The problem is the race condition.  If the tool is called by their automation during the pip upgrade, the tool can fail due to the install not being atomic - I believe pip removes the entire package first.

I've thought of various work arounds (forcing the use of flock, using symlink to atomically switch the virtualenv, wrapping the tool in a script that makes the virtualenv in a temp directory for every use...)

Is there a best practice here I'm not aware of?

Using a shared virtualenv for automation while also expecting the virtualenv's content to be (automatically) updated without the automation's knowledge/control is not a good practice - opening up for all kinds of obscure/intermittent/hard to reproduce failures.

So I wouldn't spend too much time trying to figure out a solid way of updating your package in the shared virtualenv without hitting the automation jobs - sooner or later they'll be hit by other problems stemming from the same root cause - using a shared virtualenv. Think back at the very reason(s) for using a virtualenv in the first place.

For consistent results automation should ensure that the virtualenv content is updated as needed as a preliminary step and does not (unexpectedly) change while the job is performed. Which isn't really possible with a virtualenv shared across multiple independent automation jobs. It might be possible with coordinated jobs, all sharing a common step that performs the virtualenv updates in a controlled manner to ensure virtualenv's consistency during jobs, but IMHO it's just not worthy to build such complex system, it's much simpler to not share the virtualenv and keep the jobs independent.

If, regardless of these considerations, the request for a shared virtualenv still stands then don't update the virtualenv outside (well defined and announced) maintenance windows during which execution of the automation jobs should be suspended.

As a side note it's also not a good practice to not version your packages, unless maybe if you have a very clear and easily accessible indication of the exact code version being used in each particular case (the commit SHA for example) - when problems occur you'll most likely need that info when attempting to reproduce the problem, debug it and/or verify fixes with some degree of certainty.


Sonarqube stage in jenkins pipeline fails with 403 error have a pipeline job which keeps failing on Sonar Quality Gate stage with the below error:

  [Bitbucket] Build result notified
  org.sonarqube.ws.client.HttpException: Error 403 on
  http://illinXXXX:XXXXX/api/qualitygates/project_status?analysisId=XXXXXXXX
      at org.sonarqube.ws.client.BaseResponse.failIfNotSuccessful(BaseResponse.java:34)

What's more strange, another pipeline from the same MS, is passing that stage. They both are using the same Sonarquebe user and token, and the same stage syntax.

Sonarqube version: 6.7.1 (build 35068).

Notes:

I also tried this, but it didn't work.
The logger for org.sonarqube.ws.client not working. it doesn't write anything to the log.

Problem:
The execution user/group had permissions to run  “Execute Analysis”
Solution:
Update the default template with permissions and push to projects (Bulk Apply Template)


Using password protected ssh keys If you use password-protected SSH keys, is there a way to pass the password at the command line automatically instead of manually typing it?

E.g. something like:

ssh-add <key> -p password

First you need to install sshpass.

Ubuntu/Debian: apt-get install sshpass
Fedora/CentOS: yum install sshpass

Example:

sshpass -p "YOUR_PASSWORD" ssh -o StrictHostKeyChecking=no YOUR_USERNAME@SOME_SITE.COM

Custom port example:

sshpass -p "YOUR_PASSWORD" ssh -o StrictHostKeyChecking=no YOUR_USERNAME@SOME_SITE.COM:2400

Notes:

sshpass can also read a password from a file when the -f flag is passed.

Using -f prevents the password from being visible if the ps command is executed.
The file that the password is stored in should have secure permissions.


How to store credentials that are required by an application? Everyone is saying that storing credentials in version control (git) is a bad thing. So there must be other ways to store credentials which are much better.

An application must receive credentials from somewhere to use services it depends on. These credentials are usually stored in configuration files. Manually entering each server to create that file is out of the question, since servers come and go without human intervention.

How to manage an application's credentials?

Proper management of an application's secrets has always been a challenge. New challenges came with the adoption of the cloud. There's a great OWASP presentation about the reality and challenges of storing secrets in the cloud.

You might be surprised to hear that storing secrets into the source code is one of the solution (or "architecture") presented. That's because, right now, there is no perfect architecture or way of doing this. In the end, your secrets might be encrypted... but what is guarding the encryption key? "Turtles all the way down", they said.

Every type of secret management has its strengths and weaknesses and the presentation already covers that. Instead, I'll try to go over some of the features you might be looking for in an secret (credential) management solution:

Access control: can you give write access to admins and read access to applications? Can you limit what applications are able to read (application A has access only to those secrets)?
Audit logs: required for many compliancy reports and a good way to figure if something is awry
Secure storage of the secrets: how is the solution storing the secrets? Encrypted DB? Encrypted FS? Who/what holds the encryption key, if any? How is this key used - once at startup and then securely discarded?
Keys/passwords rotation or renewal: if a secret is compromised, can you revoke it and send an updated secret to the applications? Can/should the applications pool the secret management service?
Compatibility: Some of these solutions offer tight integration with certain languages or framework. Some offer REST API. Are you interested in this?

By looking at these items, how they are important to you and how they are implemented by the solution, you will be able to choose one of the secret management service out there.


I can't find the jenkins password to unlock the website I'm trying to install Jenkins, but, in one part I need to unlock Jenkins, and I didn't find the file with the password, it doesn't exist on my pc. I tried to run this command on my terminal (Windows):
java -jar jenkins.war

But this is the result: (I'm running the terminal as Administrator)

So what do I do now?

Try finding this file: C:\Program Files (x86)\Jenkins\secrets\initialAdminPassword


In the AWS CLI how do I treat a special character as a string? I'm trying to run an AWS CLI command for a resource with a ! in its name.
I'm doing this from Centos docker instance running on my local device a MAC and if I run
echo "$SHELL"

it says it is using bash
Using echo 'test!'  works for returning the value
but testing in my CLI command it does not work
aws autoscaling put-scheduled-update-group-action --auto-scaling-group-name temp_asg --scheduled-action-name 'test!' --start-time 2020-12-06T12:00:00Z --min-size 0 --max-size 1 --desired-capacity 0 --region eu-west-1 --output json

returns
bash: !`: event not found
my AWS CLI version in the docker instance I'm running it from is
aws-cli/1.16.251

I found that if I put it in a script it sees it as a string and it's only an issue running directly from the terminal/command prompt but not an issue when running from a script with the header/shabang
#!/bin/bash
My final goal for this is for it to be part of a script so this has resolved my issue.
Im still not 100% on the issue when directly in the terminal and don't know if it's specific to the shell I'm using?


What are known useful and misleading memes in the DevOps culture? After discovering the Cattle vs Pets and 12factor apps DevOps memes which now will help me a lot with communication, I ask myself what are other known memes in the DevOps culture?

Side note: best is to establish fact why a meme is a meme e.g. more than 10K Google query results?

Useful memes:

"Wall of Confusion" from https://www.slideshare.net/littleidea/agile-infrastructure-velocity-09
C.A.M.S. from https://blog.chef.io/2010/07/16/what-devops-means-to-me/
"10+ deploys per day" from https://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr/ and https://youtu.be/LdOe18KhtT4
I.C.E. from http://radar.oreilly.com/2015/01/devops-keeps-it-cool-with-ice.html
"CI Theatre" from https://www.gocd.org/2017/05/16/its-not-CI-its-CI-theatre.html

There are some horrible memes pushed by vendors worth mentioning.

WARNING: Bad memes to use for explaining DevOps (traps for young players) -

"BI-Modal IT" by Gartner - criticised by Jez Humble here: https://continuousdelivery.com/2016/04/the-flaw-at-the-heart-of-bimodal-it/
"Multi-Speed IT" by Accenture https://www.accenture.com/us-en/insight-calibrating-multi-speed-it - (same as bimodal, but different name because of copyright - asked about here - What is good and/or bad about Accenture's Multi-speed IT approach to DevOps?
"The Three Ways" by Gene Kim https://itrevolution.com/the-three-ways-principles-underpinning-devops/ - the comments on the article include plenty of criticism


Automate Helm's tiller installation with Terraform I need to be able to generate disposable kubernetes environments using terraform.  These environments should use helm for application deployment and ideally have tiller installed during the terraform resource provision.

What is the best way of automating this?  Should I "copy" the deployment generated by helm init and translate it to terraform lingo?  Is there any other solution more elegant than this?

I tried to explicitly declare the terraform's helm provider block, but without any real chart deployment it does not bother into installing the tiller service.

How I see there is two options (from my exp):

Use configuration for provider.
https://www.terraform.io/docs/providers/helm/index.html, there is option to install tiller.
Use prefabricated modules, for example:
https://github.com/sagikazarmark/terraform-tiller
https://registry.terraform.io/modules/Vidimensional/tiller/kubernetes/0.0.1


What does a "DevOps Transformation" contain? Some consulting companies are promoting a service called "DevOps Transformation". Multiple big companies are talking about the subject at conferences and meetups around the world.

What does such "DevOps Transformation" entail?
How does it look like in actionable terms, both for successful transformations, and failed ones.

I need to put my answer to this question in the context of what DevOps is, more specifically within the DevOps transformations I have been part of, DevOps is the ownership of the full Software Development Lifecycle. All of the practices in the chart are an important part of DevOps, and they enable and underpin both Systems Thinking and Amplification of Feedback Loops.

However, the key differentiator between CI/CD and DevOps is the actual operation of software in a production environment, where it can deliver value to its customers and the business it serves.

As a consultant participating in or leading a DevOps transformation I have the following aspects in the front of my mind:

Culture: As Dave quite rightly pointed out a Culture of Continuous Experimentation and Learning is critical to the success of any transformation.  From a DevOps perspective this comes down to how do we engender a culture that is supportive of the chosen DevOps model, this model could be "You Build It, You Run It" or it could be more along the lines of Google's Site Reliability Engineering practice.
Operating Model: This is the part of the business proposition that articulates how the organisation will be delivering value, generally by articulating the People, Process and Tools used tied together at a high level.  Without an operating model, you have no blueprint for the way the organisation adopt the practices that the culture defines, this, in turn, leads to a lack of clarity and divergent behaviours.
C-Level Aircover: It is often our job as consultants working within transformation programmes to make radical changes to the way business works.  You are going to upset people, and some people won't like the changes - it is important that you have "air cover" from above to change things and move forwards.

Once the high level is in place it is important to find something you can realistically deliver:

Start as small as possible, ideally, once you have some people who understand the culture, a sketch of an operating model and buy-in from the executives create the "Minimum Viable Project", don't attempt to boil the ocean by introducing DevOps to an audience of thousands.  Set an achievable goal:

Automate the creation of the infrastructure from Product X.
Automate the delivery of Product X into Azure across all environments.
Hand-back support from outsourcer Y to a development team in London.
Create a set of tests around our riskiest feature and run them in continuous integration.

Great you have some success under your belt now it is time to start baking this into more teams, add another couple of teams into the mix and get them up and running.  Don't be afraid to offer "White Glove Support" at first to assist them in the transition; they will need a lot of hand-holding over the coming weeks and months.
Now you have several early adopters following a new way of working; you have some critical mass, it is time to start evangelising the work you are doing with a wider audience:

Hold regular show-and-tell sessions ask the early adopters to demonstrate how successful they have been.
Offer drop-in sessions to allow other parts of the organisation to explore how they can come on board with your team.
Enable the creation of Communities of Practice focusing on specific disciplines: Continuous Deployment, Automated Testing, Business Communication, Risk Management, Monitoring and Alerting, etc.

Stay the course and close off the transformation by onboarding the rest of the organisation.  Understand the relationship between the Gartner Hype Cycle and the Adoption Lifecycle.  Prepare for the Transformation Programme to fall into the "Trough of Disillusionment", stay the course and keep the end goal in sight.

For a deeper exploration of the final point read Geoffrey A. Moore's Crossing the Chasm. I could quite literally write a book about how to deliver a DevOps transformation, however by the time I had finished it there would probably be no more DevOps transformation work for me to do.


How to pull all alternative tags of a Docker image? I administer a GitLab instance with a build pipeline. All components are encapsulated in Docker images from the official GitLab maintainer.

Whenever I update - usually once a week - I need to check whether the gitLab/gitlab-runner-helper still works for the current latest version of GitLab. This can only be checked by executing a pipeline. If it does not work, the log tells me exactly what image it needs an I proceed to pull it.

The image in question is also tagged with a latest tag, which I cannot use, due to the hard dependency to the non-volatile tag.

$docker image ls
REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
gitlab/gitlab-runner-helper   x86_64-8af42251     1ee5a99eba5f        20 hours ago        43.7MB
gitlab/gitlab-runner-helper   x86_64-latest       1ee5a99eba5f        20 hours ago        43.7MB

To automate my update process, I'd like to know, how I could pull the latest image with all alternative tags?

The man page of Docker pull says, there is a --all-tags option, to load any tagged image from the repository, but this cannot be combined with a tag.

You can list all alternative image using the reg tool.


How do i specify the path of a Dockerfile using Blue Ocean I want my pipeline created using Blue Ocean (Jenkins) to build with a dockerfile that is not in the root of the git repo. How do I achieve this?

Our git repo is a collection of sub-projects with each containing it's own dockerfile. So we cannot place them in the root directory.

From the official instructions on building Docker images with Jenkins Pipeline:

  The build() method builds the Dockerfile in the current directory
  by default. This can be overridden by providing a directory path
  containing a Dockerfile as the second argument of the build()
  method, for example:

node {
    checkout scm
    def testImage = docker.build("test-image", "./dockerfiles/test")

    testImage.inside {
        sh 'make test'
    }
}

I'm not sure if you can create a Pipeline (i.e. Jenkinsfile) with this exact configuration through the Blue Ocean UI, but I would imagine you can.  However, I personally have found using Blue Ocean to create pipelines to be finicky at best, so I would recommend going over a Jenkinsfile created by Blue Ocean and making tweaks and adjustments by hand anyway.  This would give you the opportunity to adjust the Dockerfile location even if you aren't able to do it via Blue Ocean.


Why should you do port mapping for docker containers? I have been using docker for a few months and I am just a developer not a DevOps or a networking person. However, I came across a docker-compose which maps an external port to an internal port. Something like in this compose file
where it says:

ports:
  - 8080:80

My question is WHY? Why do we need to map an external port to internal port. Some of the explanation that I came across was that if you want to not expose a particular port to the user but use it internally. That also begs the same question: what would be a real life example of someone wanting to do this and why?

TL;DR

There are a number of use cases for port mapping, but for DevOps at scale the primary reason is generally to enable mapping well-known service ports to available ports on the host. This matters when you're running large numbers of containers that use the same port by default, and you don't want to manually assign or track alternative port numbers.

A Very Short Port Primer

As a general rule, a port can only map to a single service or process on each host (multiplexing ports and multi-port services are something of an exception). Only 65,536 ports are available for services to bind, with the lowest 1,024 generally reserved for binding by the root user. Services also typically bind to well-known ports such as 22, 53, or 5432 to make the service easy to find. All of these issues matter, but it's often the last issue that typically concerns Docker hosts most.

Mapping Container Ports

Imagine that you have multiple PostgreSQL containers on a single host. By default, each wants to bind to port 5432 as its default. While you could certainly modify each container or run-command to bind the container's service to a unique host port, this quickly becomes a hassle at scale.

Instead, Docker and other container managers make it easy to map ports between the host OS and the container. For example:

# launch three PostgreSQL instances
for i in {1..3}; do
    docker run --rm -d -P postgres:alpine
done

# show port mappings for each container
docker container ls -q --filter="ancestor=postgres:alpine" |
    xargs -n1 docker port
5432/tcp -> 0.0.0.0:32773
5432/tcp -> 0.0.0.0:32772
5432/tcp -> 0.0.0.0:32771

This shows that you have three instances of PostgreSQL, all happily listening on the default port of 5432 inside their containers. However, each instance is listening on a different port (32771, 32772, and 32773) on the Docker host!

At scale, you would typically use DNS, autodiscovery, linking, or container networking to help clients and applications find the right PostgreSQL instance to connect with. With just a few instances running, parsing docker ps may be sufficient for your needs. Your specific use case may vary.