Jenkins Pipeline job; proper quoting for slackSend step This is probably 100% user error, but I can't get a pipeline job to use a variable as the channel name in a slackSend step.

This uses both the Slack plugin and the Build User Vars Plugin. I'm trying to use BUILD_USER_FIRST_NAME to identify which channel to Slack back to (ie, the one starting the build should be the one to get the message).

Got halfway there with this article on how to use the build user plugin in a pipeline job, but substituting the variable is tripping me up.

The relevant part of the job looks like this;

stage ('message') {

    wrap([$class: 'BuildUser']) {

        sh "printf '%s' ${BUILD_USER_FIRST_NAME} > name.txt"

    }

    archiveArtifacts artifacts: 'name.txt'
    name = readFile('name.txt')
    echo "name is $name" //works fine, I get Alex

    if ( "$output" != null ) {
        slackSend (channel: '@$name', color: '#36A64F', message: "Job succeeded")
    } else {
        slackSend (channel: '@$name', color: '#36A64F', message: "Job failed")
    }

}

I've tried the following in place of the channel bit, all have failed with indeterminate Slack post may have failed. Response: Invalid channel specified errors.

channel: '@${name}',
channel: '@" + ${name} + "',
channel: '@"${name}"',

Expected output is that it would resolve to channel: '@Alex',

What's the proper syntax?

In Groovy you have to use double quotes to get string interpolation:

    if ( "$output" != null ) {
        slackSend (channel: "@${name}", color: '#36A64F', message: "Job succeeded")
    } else {
        slackSend (channel: "@${name}", color: '#36A64F', message: "Job failed")
    }

You can also probably do something like this to avoid it entirely and have less duplication:

name = '@' + readFile('name.txt')

slackSend (channel: name //etc.


Is there a php (or composer) command to change upload_max_filesize && post_max_size from terminal? Is there a php (or composer) terminal command to change the php.ini variables upload_max_filesize and post_max_size from terminal to some value directly from terminal, without sed or awk operations?

The reason; having a more a version-agnostic and more backward-compatible directive in my server-environment-establishment script, to change the values of these two variables.

Update

If I'm not wrong utilizing the confd CM (supported in PHP >= 5.0) should be good (I can't test right now):

1. Create something like /etc/php7/conf.d/local.ini.

2. Add to it:

upload_max_filesize = 2000M
post_max_size = 2000M

The reason; having a more a version-agnostic and more
  backward-compatible directive in my server-environment-establishment
  script, to change the values of these two variables.

If you're trying to configure a server, you should use a configuration management system (Ansible, Puppet, Chef, Salt) or bake images for immutable infrastructure; trying to hack in pieces of configuration via PHP or bash or whatever will just make you unhappy.


Reg uploading additional files in docker hub Is there any possibilities in keeping additional files in the docker hub account along with docker images.

There is description page, but I want to maintain additional files for each image I planning to upload. Currently bundling the files in the image and retrieving it through creating container from it. It would be great if that option available.

Short answer: No.

Long answer: Docker Hub will keep the image. Anything else associated with the image needs to be kept in the source repository linked to the image in the registry.


PRs not automatically getting trigger in Jenkins github organization We are using Github for source control and Jenkins for our builds.

In Jenkins, we are using GitHub Organization Folder plugin along with Multibranch pipeline plugin. This causes the repository to show up in Jenkins, and each repository has a Jenkinsfile in it. And each PR(in github) has a status check, which checks if the build of the PR is good or not.

Everything works fine, except for the fact that once a PR is opened or updated, the PR is not built in Jenkins automatically. You have to manually go to Jenkins and trigger the PR build.

As far as master branch is concerned, it works like a charm. If the master branch is updated, Jenkins automatically triggers the build.

I have attached the screen shot of the branch configuration in Jenkins for the GitHub Organization.
So my question is what should be done for PR opening or updating, that would cause Jenkins to start build automatically.

It sounds like maybe you need to configure a webhook for git if you are using the git plugin, but I think that may just build the master branch. I haven't used git much, so you might be able to use the web hook to have it build the pull request - I don't know.

Instead, you might want to check out the pull request builder plugin and see if that does what you are looking for.


What are best practices to solve `All files/directories will have '-rwxr-xr-x' permissions` when building docker images on Windows? When a docker image is built on a Windows host the following message is returned:

SECURITY WARNING: You are building a Docker image from Windows against a non-Windows
Docker host. All files and directories added to build context will have '-rwxr-xr-x'
permissions. It is recommended to double check and reset permissions for sensitive files
and directories.

It is not currently possible to disable that warning, see discussions on Issue 20397:

  That warning was added, because the Windows filesystem does not have
  an option to mark a file as 'executable'. Building a linux image from
  a Windows machine would therefore break the image if a file has to be
  marked executable.

  For that reason, files are marked executable by default when building
  from a windows client; the warning is there so that you are notified
  of that, and (if needed), modify the Dockerfile to change/remove the
  executable bit afterwards.

In a sense this also means it's practically impossible to actually solve the issue.

What maybe can help a bit is moving the error message from stderr to stdout, released in docker 17.04. See Issue 29856.


Force Jenkins to reschedule parallel pipeline step as soon as node goes offline Exactly this. In our environment, we're using GCE preemptible nodes and the Jenkins GCE plugin. As a consequence of this, it's highly likely (even expected) that a node will go away during the course of a build. The problem is, the Jenkins GCE plugin just marks that node as "offline", but a parallel job "part" that was assigned an executor on that node will just keep churning away there, seemingly forever, apparently in the hope that the node will come back.

The question I have, then, is simple:

Is it possible to force Jenkins to reschedule (NOT fail) a parallel build "part" as soon as the node it's executing on goes offline? If so, how is this done?

There are two approaches to this, neither of them, in my opinion, are especially good as they don't strictly "NOT Fail" the build:

Use a plugin such as Naginator which can be configured to restart the build until it passes.  This can be quite frustrating as you need to differentiate between ephemeral/preemptable virtual machines being terminated and a truly failed build.
Move to an orchestrator model that manages individual jobs and restarts them based upon a set of rules.

Perhaps a better solution would be to develop a Jenkins Plugin similar to the EC2 Fleet Plugin which is responsible for spinning up and maintaining Spot Instances (i.e. similar to GCEs Preemptable Instances) resubmitting jobs when they fail due to instance reclamation.


Jenkins pipeline running using containers on an orchestration platform I have a specific implementation of jenkins on docker.

My requirement is to start slave containers on demand and only when they are needed and then stop them after the pipeline has ran.

I have the jenkins master instantiated as a container, then using the docker socket I am able to run my pipeline in one or more other containers containers provisioned by jenkins, that I have specifically built for that purpose following the implementation described in the jenkins book.

master container - https://github.com/YiannisH/containerised-jenkins/blob/master/docker/master/Dockerfile
slave container - https://github.com/YiannisH/containerised-jenkins/blob/master/docker/slave/Dockerfile

The above mentioned implementation is implemented on one EC2 machine acting as a docker host.

However, that implementation has the flaw that is not highly available and I am only able to use the resources available on on server.

How can I implement this on a swarm cluster where I still have just one master jenkins container starting slave containers from the pipeline on a swarm cluster?

I cannot seem to find something similar after extensive searching, but I am sure it is in some way possible.

You can run your application on a particular node of the cluster by defining it in the application level. For this, you need to add the "placement " parameter in the yaml file and specify the node's name to it, then it will deploy the container on that node.

Secondly, what type of scheduling have you implemented on Swarm CLuster? Let me clear your thought, there three of them: Spread, Bin-pack, and Random. By default, it is Spread, which identifies a number of containers on each node and deploys the new container on the node that has a small number of containers. Check the rest yourself.

Both of these points will solve your problem.


Can I run a Ubuntu 16 Docker image on a Ubuntu 18 host? Will I need to install hardware drivers already available to the host? We have some Docker images built from Ubuntu 16.04 base image, and we've been running them on hosts running Ubuntu 16.04; these hosts are laptop computers from different makes. However, we have had some problems finding drivers and we wind up installing them manually instead of having a smooth and seamless Ubuntu installation.

We think that it could be an option to use Ubuntu 18.04 as the OS in the hosts, as this newer versions has better support for newer PCs.

The question is: suppose that the host running Ubuntu 18.04 has "built-in" drivers for, say, the WiFi card, but those are not automatically available for Ubuntu 16.04. Will we need to anyways install those drivers manually in the Ubuntu 16.04 Docker image or will it just run OK since they are correctly installed in the host?

I cannot get it from Docker's own system overview.

Docker containers do not use hardware drivers. A containers network connection is abstracted by Docker bridge.

I highly recommend you seek some free tutorial videos or check out Linux Academy to better understand Docker. I promise it will give you great insights.


AWS API Gateway: 403 forbidden with disabled API key; error from cloudfront How can happen that with a Lambda/API Gateway combination, if no explicit Cloud Front configuration has been done, in an error response header they mention CloudFront?
header: x-amzn-ErrorType: ForbiddenException
header: x-amz-apigw-id: P3mmbT_DREF8bg=
header: X-Cache: Error from cloudfront

Actual problem: with a disabled API key, API keeps responding "forbidden".
UPDATE. As it seems the problem is not related to the Edge API/CloudFront, as the problem with disabled key persists also on changing API type to regional.
UPDATE. I can now disable API key checking as suggested by @Harish i.e. it's not about disabling the API key itself but telling the API whether it's required or not.
What is yet open though is why if created by Zappa, there is no option to change this setting in the UI, because there are not single methods.

When you deploy an edge-optimized API, API Gateway sets up an Amazon CloudFront distribution and a DNS record to map the API domain name to the CloudFront distribution domain name. Requests for the API are then routed to API Gateway through the mapped CloudFront distribution.

Source: API Gateway documentation — Edge-optimized custom domain names.

To allow calls to a method of a resource in your API without API key, set its API Key Required setting to false:


How can I access directories inside a docker container from outside (not the reverse!)? I am trying to setup plug-and-play docker images for my development and support team members - however I am running into some problems because I want them to be able to use the editor of their choice!

There are two critical folders that are part of the build (eggs and src) that needs to be exposed to the end users so that they can edit it in any editor of their choice.

All the options to edit container files in host machine talks about mounting a fresh folder from the outside; but like I said, my src and egg folders are part of the build, so I can't let them mount a fresh folder during docker run.

I thought of about using volumes, but here are the questions I have on this approach:

1) Do we usually put build stuff that are actually part of the container into volumes? The container has no meaning without these volumes.

2) Can end users take a copy of these volumes and just modify it according to their convenience without the master copy of these volumes? (Like Image vs Container concept?)

All my problems will be solved if I can expose folders inside my container to the outside (not the reverse), but that seems really tricky to achieve!

PS: I haven't research enough on volumes but I thought I could ask this question anyway.

Just to complete the workflow, our development/support team usually identifies and fixes these problems in their machine, sends a PR to the Git system which then gets merged etc.

I am trying to use Docker to solve the problem of not re-building the entire web-app in each developers machine running multiple OSes etc.

I think you may be using the wrong tool for the job.  Docker is not designed to be a version control system, it is designed for containerization.  Furthermore, your question makes it sound like the development and support team members are constantly changing the files that are essential to the makeup of the image.  This makes using a volume pointless as the Docker image will always be changing.

If I'm understanding what you are trying to achieve (people should be able to modify the src and eggs folders with whatever editor they want), then you may want to explore adopting a CI/CD workflow:

Committed changes to the src and egg folders trigger a new image to be built.
The new image is pushed to a Docker repository.
Team members can then pull the updated image.

As for your two specific questions:

  Do we usually put build stuff that are actually part of the container into volumes? The container has no meaning without these volumes.

Not usually.  Generally if I have to make changes to a container I would remote into it and then commit the changes.

  Can end users take a copy of these volumes and just modify it according to their convenience without the master copy of these volumes? (Like Image vs Container concept?)

This is sounding like a version control system which Docker is not designed for.  You can have a shared volume that users have access to, but the use case is for persisting data outside of containers (e.g. SQL database data).


Docker not found in circleci config Why is docker not found? Do I need to use a different docker image than debian:latest? I want the latest debian image so does that mean I need to manually install Docker?

orbs:
  aws-ecr: circleci/aws-ecr@6.2.0
version: 2.1

workflows:
  # Build every commit on every branch without pushing to ECR
  build_image:
    jobs:
      - build_image

jobs:
  build_image:
    docker:
      - image: debian:latest
    steps:
      - aws-ecr/build-image:
          account-url: AWS_ACCOUNT_URL
          repo: node

Try adding,- steup_remote_docker before - aws-ecr/build-image:


How to communicate queue-based processing delays to non-technical team members? I am responsible for a set of SQS queue-processing jobs with a scaling policy on the ApproximateNumberOfMessagesVisible CloudWatch metric.  These jobs can fail to keep up with the amount of messages sent for any number of reasons:

Service degradation reduces capacity of messages able to be processed.
AutoScaling max limit reached while queue depth continues to rise.
S3 Outage impacts other dependent AWS services (AutoScaling service) that the queue processing job uses to keep up with demand.

When discussing outages with non-technical team members, I would like to communicate specific delays of queue processing that can translate into customer-visible degradations.  How can I do this with SQS queues?

As with any outage communications, a non-technical reader will be primarily looking to understand:

How long was it?
How bad was it?

Amazon CloudWatch metrics provides the following metrics for SQS queues that can help answer these questions:

NumberOfMessagesSent: The number of messages added to a queue.
NumberOfMessagesReceived: The number of messages returned by calls to the ReceiveMessage API action.
ApproximateNumberOfMessagesVisible: The number of messages available for retrieval from the queue.

When graphed correctly, these metrics can be powerful visual aides in describing queue processing delays. Here are a couple of examples from an outage I experienced where the job's capacity to process queue messages was severely degraded:

NumberOfMessagesSent & NumberOfMessagesReceived

Graph Type: Line Graph
Statistic: Sum
Period: 5-minutes

This graphs the contrast between messages sent and received, which helps isolate which processing component is responsible for the delay. In this graph, the received metric drops dramatically while the sent metric continues on its normal trend, so we can infer that the issue lies in the queue reading component instead of the queue writing component.

Does this answer how long/how bad event was? Yes; describes processes impacted over time.

NumberOfMessagesReceived & ApproximateNumberOfMessagesVisible

Graph Type: Stacked Area Graph
Statistic: Sum
Period: 5-minutes

This graphs the queue depth on top of messages received, which helps show how far the queue backed up and how it recovered. In this graph, we can see that the queue depth backed up dramatically while the queue reading component was having issues, and began recovering when the queue reading component began reading messages again.

Does this answer how long/how bad event was? Yes; describes messages impacted over time.

Graphing Discussion

In both graphs, queue processing can generally be considered healthy when the lines overlap, and unhealthy when the lines diverge.  This is an easy pattern to teach to a non-technical team member, and can help them to quickly dissemenate where and how there are problems when presented with these graphs.

To further communicate specific points in graphs, you can simply annotate them:

Graphing Tips:

Label units and axes.
Use consistent colors for matching metrics across graphs.  Note that NumberOfMessagesReceived is orange in both graphs; this will help visualize the same metric across different graphs.
Vertically align graphs that describe similar metrics so that they are easier to compare across time.

Note: I've formatted these graphs for presentation on StackExchange, so these aren't necessarily how I would present them in an outage post-mortem. I have explicitly removed values from the left axis here to obscure them from StackExchange; you'll want to keep them in your post-mortems.

Additional tips

Empower Your Team: After training your team members to read these graphs, there is no reason keep them hidden away. Consider setting up a CloudWatch Dashboard and giving your non-technical team members read-only IAM access to CloudWatch, so that they can view these graphs any time.
Set Up Notifications: Consider setting up Cloudwatch Alarms based on the ApproximateNumberOfMessagesVisible metric if it exceeds some agreed-upon high value, and subscribe team members to notify them of potential issues. Cloudwatch Alarms have description fields that are sent along with the notification emails -- make sure to include a human-readable description to help your non-technical members disseminate the alarm.
Explore Other Data: Per Evgeny's comment, explore other data beyond what CloudWatch provides and think about how you can convey that data to your team.  His example of using message lifetime in the queue to create a histogram is a great example of this creative thinking, and can be accomplished by logging both the message send and message receive times in your application.  You can get the message Sent Timestamp via the SentTimeStamp Attribute on each queue message of the ReceiveMessage API response.  More details here.


How to use stored certificate in Jenkins declarative pipeline? I have a stored certificate in Jenkins, and I need to use it in a pipelinebuild job, but the problem is it appears the Pipeline can't use certificates:

trying to use:

    environment {
        KEY = credentials('mycert')
    }

results in:

ERROR: No suitable binding handler could be found for type com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl. Supported types are StandardUsernamePasswordCredentials,FileCredentials,StringCredentials,SSHUserPrivateKey.

How do I get around this and use that stored cert in a pipeline? There has to be a way otherwise why would storing certs as credentials even be possible.. would be useless to store something without ever being able to use it.

I usually set the credential NAME into an environment variable

environment {
    CERT = 'mycert'
}

Then I can use that name where ever I need it

withCredentials([usernamePassword(credentialsId: "${env.CERT}",
                                  passwordVariable: 'CERT_PASSWORD',
                                  usernameVariable: 'CERT_USER')]) {
    // Do things needing environment variables
}

or

checkout(poll: false,
         scm: [
               $class: 'GitSCM',
               userRemoteConfigs: [[credentialsId: "${env.CERT}",
                                   url: 'https://code.example.org/somerepo']]
               ])


How to copy files from a different linux server folder to Kubernetes pod I have multiple files in a  different linux server folder and I want to create a cron job and run daily in kubernetes that should copy the files to a pod network folder.

I am researching a lot on this, but non of them gave a me clue to do this experiment. Is it we need to create Persistent volume claim against linux folder? Please help to accomplish this.

I have multiple files in a different linux server folder and I want to create a cron job and run daily in kubernetes that should copy the files to a pod network folder.

I am unaware how to perform the task as you have stated.

How to solve your problem? With the belief your "POD network folder" is a SMB, NFS, other type mounted resource;

I would implement one of the following, either will work the same in the end:

Establish a connection directly from the Linux Host (that presently has the file(s) to be copied) to the Networked Folder. Use SCP or Rsync to perform the copy.
Using SCP or Rsync copy the file(s) directly from the Linux Host homing the files to the IP Address of the POD. Given the POD has a standing connection to the Hosted Folder. e.g. NFS, SMB, other mount.


What is the difference between Container Engine Software and Container Management Software What is the difference between Container Engine Software and Container Management Software ?. Docker/kubernetes are managements softwares whereas Apache Mesos/Rocket are Container Engine softwares. Please explain.

I see a lot of confusion in the question. Docker is a containerization software whereas Kubernetes and Mesos are container management softwares.
While Kubernetes is a software for cluster management of containers(not necessarily for docker) and microservices, Apache Mesos is an orchestration technology for cluster management softwares like Kubernetes.

All these three types of technology are aligned to work together.

In layman terms, you (can) run Docker within Kubernetes which you (can) run within Mesos.

Please have a read here for more explaination.


Is there a website that shows the popularity of CI/CD tools? There are websites that indicate the popularity of programming languages and open source tools. Is there something comparable for CI/CD tools as well?

I normally use stackshare. It doesn't show you usage as reported by the people running the tools but it has a decent community size and seems to be gaining more use rather than less.

It lets you compare tool features, community popularity and shows you who else is using those tools in their stack if that matters to you.


Constructing Dependencies from a Pillar in Saltstack I'm trying to create a dummy state in Salt to pull dependencies from a list derived from a pillar.

In my top.sls file, I have:

base:
  '*':
    - components

Then, in components.sls, I have:

{% if 'components' in pillar.items() %}
include:
{% for component in pillar.get('components',[]) %}
  - {{ component }}
{% endfor %}
{% endif %}

{% if 'components' in pillar.items() %}
components:
  require:
{% for component in pillar.get('components',[]) %}
    - {{ component }}
{% endfor %}
{% endif %}

When I check the pillar contents, I see:

$ salt my-minion-id pillar.items

my-minion-id:
    ----------
    components:
        - a-dependency-name

But when I try to see the set of states, I see:

$ salt my-minion-id state.show_sls components

my-minion-id:
    ----------

...and that's it.

My ultimate goal, here, is to treat the minion as hostile, so I'm trying to pull role details out of pillars instead of assigning roles in my file_root's top.sls. It's my understanding that every minion has full access to the contents of the file_root, and I don't want a hostile minion to know the specific firewall or services of any unrelated node.

Edit:

I discovered that my pillar structure didn't support pillar merging, as it used lists, so I switched to a pillar-structure that used no-value dicts:

my-minion-id:
    ----------
    components:
        ----------
        a-dependency-name:
            None

This templated SLS file works splendidly:

{% if 'components' in salt.pillar.items() %}

include:
{% for component in salt.pillar.get('components').keys() %}
  - {{ component }}
{% endfor %}

{% endif %}

However, it requires changing my pillar key structure. From the original question, you see the structure as:

my-minion-id:
    ----------
    components:
        - a-dependency-name

Instead, it needs to be:

my-minion-id:
    ----------
    components:
        ----------
        a-dependency-name:
            None

Note that a-dependency-name is now a dict, and has a single key/value, None. Thus the pillar SLS file needs to change from

components:
  - a-dependency-name

to

components:
  a-dependency-name: ~

While you can get away with not using no-value dictionaries (and thus get rid of the .keys() in the template) and use lists instead, if you do that, you can't merge the components from multiple different pillars; each pillar Salt applies will override the components key from the previous, and the last pillar read will win. If you want to merge pillars (we are talking about role-based assignment, here), this would appear to be the necessary construct.

With that, the output of salt my-minion-id state.show_sls components is then correct:

$ salt my-minion-id state.show_sls components
my-minion-id:
    ----------
    a-dependency-name:
      ----------
      ...

pillar.items() in Jinja templates turns out not to be quite equivalent to salt.pillar.items(); if you try using pillar.ls(), for example, you may see the following error:

Rendering SLS 'base:components' failed: Jinja variable 'salt.pillar object' has no attribute 'ls'

Whether that means one should avoid the implicit salt. prefix shortcut Jinja provides, or instead use a construct like {% if pillar['components'] is defined %} (thanks, @brousch, for the advice), I can't say.


Does Jenkins support cloning git submodules with "git@" (not "https") URLs? Had some issues with Jenkins when trying to checkout a project with Git nested submodules. It failed with a permission error when cloning the submodules:

hudson.plugins.git.GitException: Command "git submodule update --init --recursive some/path" returned status code 128:
stdout: Submodule path 'some/path': checked out '34e663c11661b5153076c1d624ead25137df0783'

stderr: Cloning into 'some/other/path'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:foo/foo.git' into submodule path 'some/other/path' failed
Failed to recurse into submodule path 'some/path'

In the Jenkins job, checking out submodules is enabled:
In the Git plugin under Advanced sub-modules behaviours, the options Recursively update submodules and Use credentials from default remote of parent repository are enabled.
All git repositories are private Github repositories, but the Jenkins user has read access to them.

The only way to get it working as by replacing git@... URLs by https://.... For instance, this here would fail

[submodule "foo"]
    path = some/path
    url = git@github.com:foo/foo.git

... but this would work

[submodule "foo"]
    path = some/path
    url = https://github.com/foo/foo.git

I cannot rule out that there is a misconfiguration on our end. But still I have some generic understanding question:

Does the Jenkins Git plugin support git@ URLs? Or is it a known limitation that you need to use https://... URLs?
Why does the choice of the protocol make a difference on Jenkins? Is it because git@ assumes that SSH keys of Jenkins were added to the Github repository?

As a reference, it works for our system, submodules over SSH. We use deploy keys for this purpose. In Gitlab they can be found under Settings --> Repository --> Deploy Keys. The used key has to be marked as Enable in each used Git repository (main and submodules).


Any tools available for rollout planning / checklist I deploy applications on servers. While the process itself is automated but there are lot of manual steps involved surrounding the automation due to various reasons.

I manage the manual procedure in Excel. But this is getting cumbersome.

I am wondering if there are any tool where the procedure can be described and easy to manage.

Thanks

Document the functional process

When I create a new project and want to deploy it I always start to right down the manual steps in a README, but one could also create a txt file or sheet like you did.

Feel the pain

At a certain moment when I have deployed the project manually the pain of certain steps will become clear. For example, I add per step how much time it takes.

Automate the high ROI steps

If for example a certain step takes half an hour an one deploys twice a week, one could investigate how long it will take if one automates this step (investment) and what time it will save (return). One could decide to create issue tickets and automate steps every sprint.

Example deployment plan

For one of my current projects I have to deploy multiple microservices. I will not provide all details, but initially the plan was two pages long. Now I only use a Continuous Integration/Deployment (CI/CD) plan and a lot of abstractions. If the deployment will fail some day I could still read the initial plan so that the functional flow is still clear.

Configuration as Code

Another benefit of automating the manual step is that one could save the code in a Git repository as a pipeline script. When another person reads the scripts then it becomes clear how the project is deployed and one could also trace who made some change, what modifications were done and when.


Kubernetes user mode in production Docker daemon has to run in root mode which brings up security discussions.

How is this with Kubernetes daemon - can it run not as root?

Note: both of these products allow you to reduce the scope of kernel capabilities allowed for containers but this is not the question.

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

https://linux-audit.com/linux-capabilities-hardening-linux-binaries-by-removing-setuid/

  With Linux capabilities, you can grant certain privileges to a process
  without granting all the privileges of the root user. To add or remove
  Linux capabilities for a Container, include the capabilities field in
  the securityContext section of the Container manifest.

In summary, in k8s it is possible to assign certain privileges to a pod. Assign the privileges that are required by a certain container.

There is an open issue:

https://github.com/kubernetes/kubernetes/issues/7961


Leadership types for DevOps team? Would you say from experience, either task or relationship leadership style works better for DevOps?
Or would you go for something else in this context than Forsyth's theory?

Different types of leadership styles are required. To start with DevOps in a company, one has to apply a different leadership style compared to when it has already been implemented.

When one starts with DevOps a cultural change has to be done and engineers should get rid of, for example, the "throw it over the wall" mindset. When DevOps has been implemented, a leader should be able to communicate well with other teams to connect multiple departments in a company.

In summary, in the beginning a leadership style that motivates people to implement DevOps is required. Later communication between teams and how the team will work in the organization, i.e. more focus on external.

A leadership style that could be used to implement DevOps is a transformational leadership style.


Is it necessary to specify resource in S3 Bucket Policies? Is it necessary to specify a resource attribute within the statements that make up an S3 bucket policy? The Access Policy Language Overview seems to suggest that the resource attribute is always included, though it doesn't come straight out and say it's required. Consider this example policy shown on the language overview page.

{
  "Version": "2012-10-17",
  "Id": "ExamplePolicy01",
  "Statement": [
    {
      "Sid": "ExampleStatement01",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::Account-ID:user/Dave"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::examplebucket/*",
        "arn:aws:s3:::examplebucket"
      ]
    }
  ]
}

A bucket policy is assigned to an S3 bucket, so it seems like the policy would always be evaluated in the context of the bucket that it's assigned to. Based on that, it seems like it should be possible to omit the resource attribute.

If the resource attribute really is required, is it possible to use a wildcard arn like this? Would that carry any risks, or would it simply allow access to the bucket exactly as if the bucket was named in the policy?

      "Resource": [
        "arn:aws:s3:::*",
        "arn:aws:s3:::*/*"
      ]

(For greater clarity, I am NOT talking about IAM policies in this question.)

Yes, Resource is a required element on S3 bucket policies. If the resource is omitted, the API will respond with an error resembling this one below.

status code: 400
Error putting S3 policy: MalformedPolicy: Statement is missing required element

The reasoning is that each statement could apply to the bucket itself, the contents of the bucket, or some subset of the bucket contents, and it's up to the user to clarify. In theory AWS could provide a default that includes both the bucket and all of its contents, but that design probably wasn't chosen because it makes it too easy to accidentally make the bucket policies broader than expected.

Additionally, it is not permissible to use a wildcard in the resource element of an S3 bucket policy. Attempts to do so will yield an error such as this:

Error putting S3 policy: MalformedPolicy: Policy has invalid resource


Git feature branching strategy allows unapproved 'tag along' features Context
Our current operations consists of three servers: QA is a playground for devs to try out new features; Platform is the live customer facing application; Mock is a copy of Platform and functions as a staging area for new releases to be tested before release to Platform.
Our branching strategy follows these three servers. There is a QA branch for the QA server, Mock branch for Mock, Master for Platform. When a new feature is requested, a feature branch is branched off Master to be developed on a devs local machine. Once the feature is complete, the feature branch is merged into the QA branch for testing/approval. When release time comes around, we pick out all the approved feature branches and merge them into the Mock branch for final release testing. After release testing the Mock branch is tagged as the next version and merged into Platform.
Symptom
The main issue that we have been dealing with using this strategy is that devs who finish a feature will accidentally branch off the newly finished (or sometimes still in progress) feature branch instead of the Master branch when moving on to the next feature. This means that when approved features are added into Mock for final release review, they can often carry along unapproved features as they sit in the same feature branch.
Question
Is there an automated way to go through each feature branch to check that no other feature branches are in its history since the last release tag? I can do this manually (which I do occasionally) but that becomes cumbersome. I can also make my own script to see if any branches share the same commit since the last git tag if such automation doesn't exist.
I am not trained in DevOps - if you see a glaring issue with this dev cycle please point it out to me!

Technically I guess something like this would print out any branchs between any particular branch and your master branch:
for hash in $(git log --format=%h master..); do if git branch --contains=$hash | grep -qv $(git name-rev HEAD | sed -e 's#HEAD ##g'); then git branch --contains=$hash | grep -v $(git name-rev HEAD | sed -e 's#HEAD ##g'); fi; done

But fundamentally there is probably an underlying problem here, I'm not intimate with your work flow or the teams you work with, but it seems unlikely to me that developers would accidentally create a new feature branch rooted on another feature branch unless they actually wanted some of the code in that other feature branch, perhaps to reuse in delivering whatever the new feature might be.
It seems that you might make a lot of improvement to your process quite quickly and easily by simply ensuring that the all features that are currently being worked on to completion are included in the next release and thus cN be merged into a pre-release branch as soon as they are approved. From there any branch taken from that pre-release branch should only contain features intended to be released,but it would mean that if a developer wrote code they thought was applicable to more than one problem,then they would almost immediately be able to reuse it once merged.
Although I'm not entirely sure that I've understood the issue you are facing, it seems to me that the existence of code in two branches isn't actually a problem, it's just code reuse. If one of those branches is pulled from the release then that's fine, the code that is being reused would still need to stay?


How can continuous integration scale for very large projects/teams? Traditionally CI systems only monitor the quality of the code in an integration branch, signalling when regressions occur. Human intervention is required for repairs.

As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears—progress on the branch becoming effectively negligible.

Splitting into multiple teams each working on a separate integration branch which would be merged at a later moment might help, but greatly extends the project duration, since it simply delays the necessary integration for a later moment while adding churn/noise/technical dept related to the partial integration from individual branch merges. Costs also increase due to CI setups for each of the branches, each with their own build/QA resources, etc. And overall quality is not necessarily better.

Scalability is, at best, dubious.

Is there a method for making such large projects actually scale?

Yes, but it's by preventing the things you mention in your question as much as possible.
You are right that there are only so much developers that can work on the same code before things become unmanageable. You need someone or something that has an overview of all changes and makes sure integration works fine and regression doesn't occur. This is hard to scale if everything happens in one repository.

Before starting any work you need to separate functionality in such a way that things can be integrated later with little or no effort.
This means that you want to split functionality into parts that are independent, or at least have as little dependencies as possible.
How you do this depends very much on the product you are developing. In the past separation of functionality was done by creating different libraries. The downside to this is that you need a good method of managing those libraries. Old versions of Windows for example had a bad library management system which caused the infamous DLL Hell.
Nowadays micro-services is (becoming) a popular way to do this, although this also has it's drawbacks (e.g. communication overhead, more services to monitor)

If separation based on functionality is difficult, another option may be to separate development in time. Instead of multiple teams working simultaneously on many different features, you have 1 team that works on only 1 or a few features at a time.

When separation of work is not possible, you'll need tools and tests that make sure things align. Automated unit and integration testing can be a big help here, but ultimately there will be limits to how big a project or team is to be workable.


trigger command on registry update I'm fairly new to CI/CD so this is probably a noob question.

Anyway, here's my current setup:
* Bitbucket repo
* Registry service on AWS (ECS Registry)
* Pipeline on Bitbucket that builds a docker image on master branch update and pushes it to the registry.

So far so good, I'm missing the last part where my server (EC2 instance) listens for registry updates and executes a deployment command like docker-compose pull && etc.

What tools can your recommend me to do this?

The key words here are:

  Pipeline on Bitbucket

You already have a pipeline. This is where you would stick the deployment command.

AFAIK there is no standard tooling for registries to 'push out' any changes; you'd have to code a custom integrator that continually polls for changes and compares. This would be a lot of work. Same goes for having your servers directly listen for changes on the registry.

It would be far easiest (and is the standard) to simply have your build pipeline issue the deployment command after its successfully pushed a new image to the registry.

There are benefits to this:

your pipeline is aware of whether the push to the registry failed or passed (and you can handle that failure gracefully with retries)
your pipeline will be aware of the new registry tag
your pipeline can control exactly which tag gets pushed where, when, and how quickly
your pipeline has the same insight to whether the deployment passed or failed, and can attempt retries or perform notifications of failure


How to mark build success when one of the stages is aborted? I've a pipeline with stages where one of the stage, intermittently takes longer than expectation and hence using timeout to abort it. But if the stage is aborted, build also marked as aborted. Following is the code for pipeline -

pipeline {
    agent any

    stages {

        stage('First') {
            options {
                timeout(time: 10, unit: 'SECONDS')
            }
            steps {
                script {

                    catchError(buildResult: 'SUCCESS') {
                        echo "Executing stage I"
                        sleep 12
                    }

                }
            }
        }

        stage('Second') {
            steps {
                script {
                    echo "Executing stage II"
                }
            }
        }
    }
}

Even though the stage is marked as Aborted, I want to mark build as Success. Can you please help how I can achieve this?

I answered that question stackoveflow and I'll post the answer here:

For a Declarative pipeline (not a scripted) you would do something like this

stage('Foo') {
      steps {
            script {
                env.PROCEED_TO_DEPLOY = 1
                try {
                    timeout(time: 2, unit: 'MINUTES') {
                        // ...
                    }
                } catch (err) {
                    env.PROCEED_TO_DEPLOY = 0
                }
            }
      }
  }
  stage('Bar') {
    when {
        expression {
            env.PROCEED_TO_DEPLOY == 1
        }
    }
    //rest of your pipeline

stage "Bar" would be skipped but as for the rest of the stages the job will be marked as passed (assuming nothing wrong happened before).

As for your particular use-case, the trick is to use a try/catch block.


Unable to get value of password parameter into another parameter In Jenkins pipeline parameter say Branches of type Active Choice Reactive Parameter, I've to auto-populate branch names hence I've following command, which I can execute in parameter's Groovy script -
svn ls <url> --non-interactive --no-auth-cache --trust-server-cert --username <username>--password <password>

username can be retrieved using User.current().getId()
I've Password parameter of Password Parameter type. To use value of this parameter in Groovy script of Branches, I've added Password parameter in Referenced parameters of Branches
But I cannot see its value reflected in Branches parameter. Can you please suggest how this can be handled?
Or any other way to pass Password value or authentication details so I can execute svn command.

Could credentials binding plugin be the answer?
The idea is to add svn user and password to Jenkins credentials store and then use it in the pipeline.
withCredentials([
    usernamePassword(
        credentialsId: 'mySvnLogin',
        usernameVariable: 'USER',
        passwordVariable: 'PASSWORD')]) {
    sh '''
      svn ls <url> --non-interactive --no-auth-cache --trust-server-cert --username $USER
 --password $PASSWORD
    ''''


AWS Cloudwatch for unused accounts Is there any possibility for getting a report on unused user accounts using cloudwatch ?
Requirement : Need to get a report on the accounts which is unused for more than 30 days.

You can get all used User accounts with Cloudtrail, it would then be a text processing exercise to find the unused ones.


What is a reasonable level of coding expertise expected of an SRE? SREs and coding is a contentious topic. In most situations, SREs end up spending a lot of time on operational tasks, even while being efficiency and automation focused. This means that SREs don't spend as much time coding as a full time developer, which puts them in a very disadvantageous position as far as programming rigor is concerned.
Also, SREs are typically responsible for multiple services or applications, unlike developers who may be more focused on a small number of applications.  Gaining deep knowledge of large codebases can be a daunting task especially if the focus is diffused by interrupt-driven nature of SRE work.
To be very good in coding and be able match a developer's level of expertise in programming, SREs need to spend more time coding on a continuous basis, which may not be possible.
Also, the question arises - if an engineer is a great developer, why would he or she opt for an SRE role?  The pure development roles seem to have a better career path.
Also, the argument that SREs should spend most of their time coding and not in operations seems a little disconnected from reality. Not all operations is evil and not all code is an angel either. As someone said, "the best code is the one that was never written". Running operations efficiently and reliably is an important goal for SREs and coding can be helpful in achieving that goal, though it is not the only means to get there.
So, what is a reasonable expectation for an SRE when it comes to coding?  Is it practical to have two cadres of SREs -- development focused ones or "SWE SREs" and operations focused ones or "Operation SREs"?  Could they be a part of the same team? Is it practical to rotate people between these two roles?
I hope this question is not dismissed as opinion-based.

If an engineer is a great developer, why would he or she opt for an SRE role?
Here is a caricature of being a developer: Working in Sprints, getting a ticket with all of the BA work completed, spending 3-4 days on that problem, marking it done, getting the next ticket and so on. This is a really boring work environment.  Yes there are better environments to work in, and not all development roles fall into the scrum mediocrity trap, but it happens frequently.
DevOps is new enough that it's not yet been "solved" as a problem.  It certainly hasn't yet had all of the engineering challenge managed into separate roles, so it allows a certain type of developer a bit more freedom, working using multiple skills, in a fast paced environment. It's not for the faint hearted.
What is a reasonable expectation for an SRE when it comes to coding?
That they are a programmer that you would want on the development team. SRE is not a place for substandard developers. The development challenges in SRE are as great as that in the development space, but the space for errors is less.
Is it practical to have two cadres of SREs -- development focused ones or "SWE SREs" and operations focused ones or "Operation SREs"?
Not really. To be in SRE you must understand Dev and Ops. If thats not the case, then you are not doing SRE, you are doing something else.  It doesn't mean it's wrong, or not working for you - it could be just what you need for your environment, and you have the people that make it work. It is possible to have a specialist who isn't really a developer on the team, IF they bring an important skill to the team that the developers are not able to cover (e.g. low level networking or linux kernel knowledge), but this is done rarely.
Could they be a part of the same team?
You can place Dev and Ops into the same team.  It would make for some interesting management challenges. Please run the team in a more kanban style, ops doesn't work in sprints
Is it practical to rotate people between these two roles?
Most developers benefit from an exposure to the land of DevOps.  However, if it is a true ops environment, that timescale is probably 2-3 days at most.  Not sure most ops people would be able to assist meaningfully as a developer, but they would often make a good tester. Most ops people would not want to be in a testing role for long.


AWS S3, as an organizer see empty bucket I am an owner of organizer in on my aws account. When I enter S3 I see one folder, which is empty. But its being billed 8$ per month, so smth should be there.
Is it possible that one account, of my organization uploaded there some stuff and I dont see it ?

Does the bucket have versioning enabled? You should be able to see it in the bucket properties: Versioning: enabled. If so, when you are looking at the empty folder, at the top there is a switch "show versions", switch it on. Do you see some objects with hashes?
Try to delete them. Be careful, there are versions and "delete markers", you have to delete both in order to completely empty the bucket.
Another option: you are billed for requests and data transfers https://aws.amazon.com/s3/pricing/?nc=sn&loc=4
If something is checking the bucket rapidly, it can generate some costs.
But most often the delete versions is the reason.


Do not show all docker containers to a particular user I want to use one server to host multiple docker-containers. I want to give another user a possibility to manage new docker-containers, but I do not want to give him access to all other containers, which I started, since they may contain sensitive data.

Is there a way to create a secure configuration which doesn't require creating virtual machine?

Hi and welcome to DevOps SE!

To enable Role-Based Access Control (RBAC) for Docker Engine, you need the UCP (Universal Control Pane) available through Docker EE.

https://success.docker.com/article/rbac-example-overview

A free alternative could be also the Kubernetes module rbac.authorization.k8s.io.

Side note: To control docker commands available to a user, use AuthZPlugin for the Docker engine (there are open source implementations of this API).


Puppet cannot find epp template I am attempting to use an EPP template in Puppet 6.2.0.

I have the following file /etc/puppetlabs/code/modules/profile/manifests/linux/base/dns.pp :

class profile::linux::base::dns {
  file { '/etc/resolv.conf':
    path => '/etc/resolv.conf',
    ensure => file,
    owner => root,
    group => root,
    mode => 0644,
    content => epp('etc/resolv.conf.epp'),
}

The EPP template file is /etc/puppetlabs/code/modules/profile/templates/etc/resolv.conf.epp:

# This file is managed by Puppet. Manual modifications will be overridden.
domain <%= lookup('profile::linux::base::domainName') %>

However, the catalog does not compile with the error : 'Could not find template 'etc/resolv.conf.epp'

Why can this module not find the template, when it resides in the modules own template directory?

Ah I worked it out now.  The epp call needs the module name in it:

content => epp('moduleName/file.epp')


How do I get the find module to work in Ansible? My playbook does not create a /tmp/results.txt file, but I expect it to do so with the content of the results of the find module.

Here is my playbook:

- name: Do stuff
  become: yes
  hosts: fun
  tasks:
  - name: Delete directory
    find:
      paths: /home/jdoe/subdir
      patterns: "jj*jj"
      use_regex: yes
    register: find_results
  - shell: "echo {{ item.path }} > /tmp/results.txt"
    with_items: "{{ find_results.files }}"

A managed node with the "fun" label has this subdirectory:
/home/jdoe/subdir/jj3jj/

What is wrong with my playbook?

Remove use_regex: yes. Quoting from the doc

  If false the patterns are file globs (shell) if true they are python regexes.

If you want to use Python regex then the pattern would be probably something like this

patterns: '[j]{2}.[j]{2}'

You probably want to append to file >>

echo {{ item.path }} >> /tmp/results.txt


Is there a 5$-30$ IaaS that supports some rolling-release distro? Is there a 5$-30$ IaaS that supports some rolling-release distro?

I consider a rolling-rlease OS (like ArchLinux) upon non rolling ones or at least upon a rolling-combo such as a rolling Win10 + Vagrant + rolling-Ubuntu) to save resources (both hardware and money).

ArchLinux seems to me to be the most communal and documented one out there but I might be wrong in that assumption.

I once asked DigitalOcean if they offer any IaaS VPSs ("droplets") with ArchLInux but a staff member told me they had that option and removed it for technical conflicts with their system and services.

Linode is one of these (see linode.com). Linode offers 5$ ArchLinux and yesterday I talked with their staff and two staff members confirmed to me that they can't find any plans to stop supporting it in the future, so that's good.

According to ArchLinux wiki here, Arch Linux is a full rolling release OS which is basically "true rolling release OS":

  Linux Mint Debian Edition (LMDE) is based on Debian Stable and only
  receives updates in Mint packages and security updates. Arch is
  instead a full rolling-release distribution.

I understand that to be the highest one can go in rolling upgrades and of release as well.


Cannot install Ansible So I am creating a packer template for docker with the image php:7.0-apache.
I added a shell provisioner where I try to install ansible so I can provision the image:

{
            "type": "shell",
            "inline": [
                "apt-get -y update",
                "apt-get install -y software-properties-common",
                "apt-add-repository ppa:ansible/ansible",
                "apt-get -y update",

                "apt-get install -y sudo",
                "apt-get install -y ansible"
            ]
        },

However, I get this error. Why is that?

EDIT: After installing the gnupg package, there still seems to be issues that result in the building process to fail:

As it says, No such file or directory: 'gpg', so you probably need to install gnupg before adding the PPA repository:

apt-get install gnupg


Is there any concrete and acceptable solution for running systemd inside the docker container? I have seen many workarounds for this to run systemd inside docker containers but looks like most of them compromise the security of the container and the host. How are most people here dealing with running systemd specific stuff inside the container.

You didn't mention what distribution you're using inside your container (which would have implications w/r/t which version of systemd you're using), but the following will successfully boot a CentOS container running systemd:

docker run -it --rm \
  -e container=docker \
  --tmpfs /run \
  --tmpfs /tmp \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --cap-add SYS_ADMIN \
  centos /sbin/init

This is with Docker 17.05.0-ce; older versions may require additional flags.  Using a stock centos:7 image, your initial environment inside the container looks like this:

# systemctl status
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2017-12-04 17:47:15 UTC; 45s ago
[...]

And:

# ps -fe
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 17:47 ?        00:00:00 /sbin/init
root        16     1  0 17:47 ?        00:00:00 /usr/lib/systemd/systemd-journald
dbus        26     1  0 17:47 ?        00:00:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root        28     1  0 17:47 ?        00:00:00 /usr/lib/systemd/systemd-logind
root        30     1  0 17:47 console  00:00:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
root        32     0  0 17:47 ?        00:00:00 bash
root        58    32  0 17:48 ?        00:00:00 ps -fe

Note that I'm using --rm here not because it's necessary but because I'm terrible at cleaning things up after the fact.  It's not necessary to get the container to run.

  but looks like most of them compromise the security of the container and the host

Well, running systemd does require privileges beyond those granted to a typical Docker container (hence the --cap-add).  Whether this has security implications for your environment or not depends on what you're doing.


Which university has a good postgraduate DevOps program? I am looking for a university that has a good PhD program in theoretical DevOps topics. Especially any program that is trying to advance the beyond Deming's Theory of Profound Knowledge, Goldratt's Theory of Constraints and other related topics. Do you know a professor specifically interested in running a program like that and currently doing advanced research in this field?

I have found some companies, which do consulting in this area and try to push it forward, but I am having problems to find some program in academic setting with sufficient credibility.

I'm not really familiar with the topics you mentioned. However I went ahead to go checkout kuleuven.be, i.e its Computer Science departement (with sufficient credibility, trust me).

Here are some interesting articles in which DevOps is mentioned:

Automated workflow regression testing for multi-tenant SaaS: Integrated support in self-service configuration dashboard.

Here is part of the abstract of it:

  ... This effectively makes these tenant administrators part of the DevOps team, and in turn creates the need for the SaaS provider to provide them with Quality Assurance tool support. One such tool is a regression testing framework that allows them to make sure that a new version of a workflow can behave similarly as to a successful execution of a previous version. ...

Variability middleware for multi-tenant SaaS applications.

Here is part of the abstract of it:

  ... The focus has been mainly on (1) the pre-deployment phases of the development life cycle and (2) fine-grained (tenant-level), run-time activation of specific variants. However, with upcoming trends such as DevOps and continuous delivery and deployment, operational aspects become increasingly important. ...

Both articles have links with contact info to their authors.


Environment variable names, when my development environment has multiple projects The context is that I'm looking at doing staging/testing and production on a  kubernetes cluster.

Where I might have some code that looks like this:

doSomething({
  username: process.env.DB_USERNAME,
  password: process.env.DB_PASSWORD
})

For my various kubernetes environments, this isn't going to be a problem, as those environment variables can be set specifically for the environment.

However, for my development environment, there's a good chance that I'm going to have multiple projects, (or even different images for the same project) that all have a environment variable named DB_URL.

One solution would be to just prefix all environment variables with MY_APPLICATION_NAME_.

Another would be not permanently set environment variables, but instead to just call a script that sets them when needed. But that sounds a little messy.

I'm wondering if there's a well established way that deals with this problem.

check out https://direnv.net/ it sets environment variables based on the directory you are in.

other options are:
1.docker-compose
2.plain bash to set the variables.
3. bash source .env
4. use prefixes but you would still need to set the environment you are in via an environment variable, so back to options 1-3.

if you are going to store passwords or keys in the env variables locally that are sensitive i.e not a localhost db pass, use a secrets manager like https://pkhub.io/usecases/environments, or you could script up something with aws' secrets manager


How can I load multiple paths into the workspace during docker run? My datasets and codes are in different folders and they don't have immediate parent folder.
 I want my codes to access my datasets while using docker run. Is there a way to load both codes and datasets folder into workspace?

You can use Docker volume mounts at run time.  For example:

$ mkdir -p app/src logs
$ touch app/src/foo.c logs/bar.txt
$ docker run --rm -v $PWD/app:/app -v $PWD/logs:/app/logs alpine ls -l /app /app/src /app/logs
/app:
total 8
drwxr-xr-x    2 1000     1000          4096 May  6 23:33 logs
drwxr-xr-x    2 1000     1000          4096 May  6 23:33 src

/app/logs:
total 0
-rw-r--r--    1 1000     1000             0 May  6 23:33 bar.txt

/app/src:
total 0
-rw-r--r--    1 1000     1000             0 May  6 23:33 foo.c

This mounts two directories on the host into the docker container.  Notice in this example that inside the container, they are mount on inside the other.


Why do some components vanish from the catalog list if one component has been installed in Rancher? If one adds a component from the catalog in Rancher and one would like to add another component than the list seems to be reduced. Why do some components vanish from the catalog list if one component has been installed in Rancher?

You haven't specified anything concrete, so I can't give you a specific reason...  But catalog items can:

Be mutually exclusive (e.g. you can't use ipsec and vxlan at the same time)
Be deployed at most once in an environment
Require the environment be a specific orchestration type
Require a certain range of Rancher versions
etc


Waiting for Solr to load By current state of the art, we cannot enforce an ordered start sequence of service containers directly, neither in Docker Swarm nor in Kubernetes. (Maybe there is a standard solution for that via a specialized orchestraton service)

Now, imagine we want to have a Solr instance with considerable amount of data, and want to get it ready before other services can start their work.

What is the best way to verify this status?

It seems for example that calling and validating the expected list of loaded collections is not enough:

/solr/admin/collections?action=LIST

Other ideas?

This has worked - the following query has to return content-type application/xml:

 /solr/COLLECTION/select?q=*:*&rows=0


docker compose & azure key vault - passing secrets to a container on startup I am using docker-compose to create 5 containers that run my application.
Two containers need credentials retrieve from Azure Key Vault (web.config passwords to access 3rd party service).

As I am committing this to a public repo I do not want to store any azure credentials in plain text in the docker-compose.yaml file when starting up each container.

Is there a better way to do this?

I am thinking to use https://github.com/mariotoffia/FluentDocker to grab the secrets and dynamically create each container - but wonder if there is a better way?

Can I use Azure Key Vault with Docker?

Is it easier to use Docker secrets?

Or something else?

Found this -->  Passing secrets to a Docker container

Environment variables appear the preferred method - will need to run a prebuild script to fetch the secrets and then follow this

https://docs.docker.com/compose/environment-variables/

@Mods feel free to close as a duplicate


Jenkinsfile equivalent in Azure repos We have our source code in GitHub. We use Jenkins for CI. We had written a Jenkinsfile in the Github repos and used GitHub plugin for Jenkins. The plugin would scan the repos and then Jenkins would execute the code in Jenkinsfile for the builds. This works perfectly.

Now, we are moving our source code from GitHub to Azure repos(https://azure.microsoft.com/en-us/services/devops/repos/) purely for non technical reasons. From what I could tell, looking around, there does not seem to be an equivalent of Git hub plugin for Azure repos in Jenkins. Is this correct. Also is there a plugin in Jenkins for Azure repos.

I'm new to Azure DevOps so I may be wrong - I think you are asking about using a Jenkins plugin to scan Azure Repos for Jenkinsfiles.

If you were moving to Azure DevOps, you can store your Azure Pipeline configuration, azure-pipelines.yml,  in YAML format and this can be stored in the code repository (like a Jenkinsfile).

For further pipeline details see; Create your first pipeline and YAML schema reference.

It looks like you can configure Azure Pipelines to scan your organization repos in Azure Repos and GitHub

The GitHub documentation does warn that this feature is actively being improved - so it may change in the future.

For GitHub you need also configure Azure Pipelines access using the GitHub app:

https://github.com/apps/azure-pipelines/installations/new/permissions


Publish pipeline artifact to Slack I have a pipeline on Azure DevOps that publishes an artifact (releasenotes.txt).

It also has integration with Slack using ( https://docs.microsoft.com/en-us/azure/devops/pipelines/integrations/slack?view=azure-devops ) .

How can I send releasenotes.txt to Slack? Is it possible?

The stock Azure Pipelines app for Slack does not support that workflow of uploading artifacts to Slack. You could use this marketplace task and upload your release notes artifact with it.

Bonus points suggestion, if you are going down the release notes path I would take a look at using Richard Fennell's Generate Release Notes task in combination with the Wiki Updater task.

You can create a generalized Task Group or yaml template that you could pretty much re-use anywhere. For example in the Wiki updater task you can create a release name wiki page formatted like:
$(Build.Repository.Name)\$(Release.EnvironmentName)\$(Release.ReleaseName).md

And then in your Slack message send a generalized link to your release notes.
Release Notes Wiki link <https://collection-name.visualstudio.com/projectname/_wiki/wikis/Release-Notes?wikiVersion=GBmaster&pagePath=$(Build.Repository.Name)\$(Release.EnvironmentName)\$(Release.ReleaseName)|$(Release.EnvironmentName)\$(Release.ReleaseName)>

You could push them back into the Repo that the release originated from or a central release notes repository. Your release notes get source controlled and are nicely organized!


Is there any way to break long command into multi lines in GitlabCI I have a very long command in .gitlab-ci.yml file to ssh to jump host then use rsync to sync files from my repo to destination host (it does not have public ip so I need to access it via jump host).

Here is my .gitlab-ci:

image: ubuntu
before_script:
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client rsync git -y )'
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - echo -e "$PRIVATE_KEY" > ~/.ssh/id_rsa
  - chmod 600 ~/.ssh/id_rsa
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

SSH:
  tags:
    - mytag
  script:
  - ssh ubuntu@host "which ssh-agent || ( apt-get update -y && apt-get install openssh-client rsync git -y ) && ssh ubuntu@192.168.1.2 "mkdir -p /home/ubuntu/test" && rsync /home/ubuntu/test/.gitlab-ci.yml ubuntu@192.168.1.2:/home/ubuntu/test/.gitlab-ci.yml"

The command here is so long:

 ssh ubuntu@host "which ssh-agent || ( apt-get update -y && apt-get install openssh-client rsync git -y ) && ssh ubuntu@192.168.1.2 "mkdir -p /home/ubuntu/test" && rsync /home/ubuntu/test/.gitlab-ci.yml ubuntu@192.168.1.2:/home/ubuntu/test/.gitlab-ci.yml"

Let me explain it:

First it ssh to host then install rsync if it does not exist, then it ssh to my private server 192.168.1.2 and then sync my file named .gitlab-ci.yml (the file from my repo in my Gitlab server) to folder test

I just wonder if there is any way to break long command above to multi lines command?

Since .gitlab-ci.yml is a Yaml file, then just use it's syntax.
For example, you may use >:
image: ubuntu
before_script:
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client rsync git -y )'
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - echo -e "$PRIVATE_KEY" > ~/.ssh/id_rsa
  - chmod 600 ~/.ssh/id_rsa
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

SSH:
  tags:
    - mytag
  script: >
  - ssh ubuntu@host "which ssh-agent ||
    ( apt-get update -y && apt-get install openssh-client rsync git -y ) &&
    ssh ubuntu@192.168.1.2 "mkdir -p /home/ubuntu/test" &&
    rsync /home/ubuntu/test/.gitlab-ci.yml ubuntu@192.168.1.2:/home/ubuntu/test/.gitlab-ci.yml"

Sample usage of yaml multilines
There is a great resource about yaml multilines: YAML Multiline Strings
Also, there is a great SO answer about yaml multilines.
So let's speculate about pipeline based on these materials.
Since we have specific context (script for Gitlab CI) then we don't need all of theese 63 cases:
                      >     |            "     '     >-     >+     |-     |+
-------------------------|------|-----|-----|-----|------|------|------|------
Trailing spaces   | Kept | Kept |     |     |     | Kept | Kept | Kept | Kept
Single newline => | _    | \n   | _   | _   | _   | _    |  _   | \n   | \n
Double newline => | \n   | \n\n | \n  | \n  | \n  | \n   |  \n  | \n\n | \n\n
Final newline  => | \n   | \n   |     |     |     |      |  \n  |      | \n
Final dbl nl's => |      |      |     |     |     |      | Kept |      | Kept
In-line newlines  | No   | No   | No  | \n  | No  | No   | No   | No   | No
Spaceless newlines| No   | No   | No  | \   | No  | No   | No   | No   | No
Single quote      | '    | '    | '   | '   | ''  | '    | '    | '    | '
Double quote      | "    | "    | "   | \"  | "   | "    | "    | "    | "
Backslash         | \    | \    | \   | \\  | \   | \    | \    | \    | \
" #", ": "        | Ok   | Ok   | No  | Ok  | Ok  | Ok   | Ok   | Ok   | Ok
Can start on same | No   | No   | Yes | Yes | Yes | No   | No   | No   | No
line as key       |

Assumptions are:

we don't need trailing spaces at all
we don't need final newlines at all
we'll try to avoid quote escaping mess
Backslash is intended to combine multi-line scripts
almost doesn't matter how many spaces before the script

P.s. I'd suggest to lint your .gitlab-ci.yml using GitLab UI


What do businesses that employ multiple development teams use to implement DevOps in a structured manner? Adding DevOps practices to a single development team is a relatively simple proposition as it usually comes down to introducing either people to the team to build up a DevOps, Release Engineering or Platform Engineering capability within an organisation.

Some of the pitfalls organisations are trying to avoid are:

Inability to report progress consistently across multiple development teams, for example, if one team reports their progress in terms of the number of equally sized stories completed in 2 weeks; whereas another team reports in storypoints completed in a three-week sprint.
Inconsistent application of a practice means team members can not easily move between teams within the same organisation without learning new ways of working.
Mismatches in development cadence across multiple teams makes coordinating dependent releases difficult.

What can be employed to achieve this in a consistent manner in an organisation that has a large number of development teams, potentially in multiple separate business units?

Large enterprises will commonly adopt a delivery framework, or operating model, that has been built to support large enterprises.  From an Agile/DevOps perspective, which is my area of expertise there are three frameworks of interest:

Scaled Agile Framework (SAFe®) defines DevOps as delivering business value through the successful operation of software in an environment.  This is achieved by giving early attention to the act of "moving into production" and accomplished, in part, by integrating personnel from the operations function within the Agile Teams.
Disciplined Agile 2.X, also known as DAD, defines DevOps as "DevOps is the streamlining of the activities surrounding IT solution development and IT operations." this is achieved by building feedback loops between the development and operations functions within the business.  DAD goes on to describe Disciplined DevOps as the intersection between the following roles: Enterprise Architecture, IT Operations, Business Operations, Release Management, Data Management and Development.
Large Scaled Scrum does not specifically call out DevOps, Release Engineering or Platform Engineering; however, it does describe a methodology for scaling out Scrum in an organisation that is growing fast, integrating a DevOps specialism within these teams is both logical and well understood.

As far as I am aware there are no statistical data about the enterprise adoption of each of these frameworks and whether they have adopted DevOps practices in addition to Agile practices, however from experience many organisations appear to gravitate towards Scaled Agile Framework.

Kristof Horvath discusses the application of DevOps in these enterprise-grade agile frameworks in his article: Scaling Agile in Large Enterprises: LeSS, DAD or SAFe®?.

It is worth noting that adopting an "enterprise-grade" framework is not a requirement in an enterprise, at the end of the day Agile, DevOps and Lean are about adopting the right practices for your organisation, not the most popular ones.


How do I configure SSH keys in a Vagrant multi-machine setup? I have 4 VMs in my Vagrantfile - 3 application servers and an Ansible control host.

I only use Vagrant to create the VMs as I provision them manually from the ansible control host because I am still creating/editing the ansible scripts.

I can do vagrant ssh ansible and vagrant ssh app1/2/3 etc. but when I try to do ansible-playbook oracle.yml from the Ansible control host, SSH fails with

fatal: [192.168.60.10]: UNREACHABLE! => {"changed": false, "msg": "SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue", "unreachable": true}

I can successfully ssh from the Ansible VM to the Oracle VM using user vagrant and password vagrant.

The key parts of my Vagrantfile are:

config.ssh.insert_key = false

config.vm.define "db" do |db|
    db.vm.box = "boxcutter/ol67"
    db.vm.hostname = "oracle-vm"
    db.vm.network "forwarded_port", guest: 22, host: 2201, id: "ssh", auto_correct: false
    db.vm.network "forwarded_port", guest: 1521, host: 1521
    db.vm.network "private_network", ip: "192.168.60.10"
    db.vm.provider "virtualbox" do |v|
        v.name = "oracle-vm"
        v.linked_clone = true
        v.memory = 2048
        v.cpus = 2
    end
end

#Optional ansible control machine for Windows users
config.vm.define "ansible", autostart: false do |ansible|
    ansible.vm.box = "williamyeh/ansible"
    ansible.vm.hostname = "ansible-vm"
    ansible.vm.network "forwarded_port", guest: 22, host: 2204, id: "ssh", auto_correct: false
    ansible.vm.network "private_network", ip: "192.168.60.50"
    ansible.vm.provider "virtualbox" do |v|
        v.linked_clone = true
    end
    #Mount the project directory on the guest so we can run the playbooks from there
    ansible.vm.synced_folder ".", "/data/ansible", create: true
end

What do I need to put in the Vagrantfile to allow the Ansible VM to connect to the other VMs without requiring a password or extra manual steps after vagrant up?

This is just for development testing on a private network on developers PCs so security is not really an issue and comes second to ease of implementation and smooth user experience.

There is no general method and it might depend on how boxcutter/ol67 was packed.

The easiest method would be to define the password in the Ansible inventory file:

[oracle-vm:vars]
ansible_ssh_user=vagrant
ansible_ssh_pass=vagrant

The second method would be to leave the insecure private key configured on the oracle-vm machine and inject the private key to the ansible VM:

config.vm.provision "shell" do |s|
  ssh_insecure_key = File.readlines("#{Dir.home}/.vagrant.d/insecure_private_key").first.strip
  s.inline = <<-SHELL
    echo #{ssh_insecure_key} >> /home/vagrant/.ssh/id_rsa
    chown vagrant /home/vagrant/.ssh/id_rsa
    chmod 400 /home/vagrant/.ssh/id_rsa
  SHELL
end

Generate the key pair beforehand on the host machine, inject private key to Ansible VM, public key to Oracle's authorized_keys.
Generate the key pair on Ansible VM, copy the public key to Oracle VM using shell provisioner and inject vagrant as password for ssh-copy-id.

And the list does not end here, it depends on required security.


What is the best way to achieve DevOps in any environment, or it is always environment dependent? I want to know that is there possibility to make DevOps generic, and not environment specific, so that can be used with any project with same cases OR we can cover common cases with one time code.
Though language and environment changes but the logic remains same for most of the Test cases based on..

I know every organization follows different methodology, I am assuming here that We will use one standard methodology and One standard Development culture.

Run environment
Development Culture, etc.

I want to achieve Common DevOps environment that can be used for any project for common cases. Only special cases need to be handled separately.

Have anyone achieved this?

I know this is little broad question, but we this is DevOps related only.

The answer to this question about what is DevOps?

DevOps both involves Strategy and Tactics.

Strategy embraces building a culture centered around collaboration in your software development, technical operations and any other group or organization that has a stake in the deliverables whether it's software or service.

Examples of DevOps strategy may be breaking down silos, continuous delivery, continuous integration, etc.

Tactics are the actual implementation of the strategies.

This is the level where the use of software tools, technology and management methods are found as well as the actual "how to".  For instance, to foster "breaking down of silos" as a tactic, you build smaller "agile" teams of 5-7 "stakeholders" let's say programmers, systems admins, QA engineer, and a project manager.  To communicate among the team members, you use Slack.  Programmers and sysadmins "cross train" on each others tasks where it makes sense.  Sysadmins may assist in developing test harnesses for the programmers, use Selenium and python for testing.  Programmers learn how to use Ansible and assist the Sysadmins on what they need in the development and production environment.

Where you sound like your at is looking at all these cool tactics and tools that DevOps uses such as Docker, Ansible, Chef but your missing the understanding of Strategy...building of the culture, the "why" you want to use these tactics and tools.  That's ok, we're here to learn.

There is no "right" answer on what exactly DevOps is or how you implement it, but in all the talks and examples of DevOps shops and organization it boils down to the strategy of building a collaborating organization and focusing on a purpose then put it into action as tactics.


How to add shell script to jenkins freestyle job I have the below shell script:

du -sh /bbhome/shared/data/repositories/* |sort -h |tail -20 |
while IFS= read -r line;do
        DIR=`echo $line | awk '{print$2}'`
        Rep=`cat $DIR/repository-config |grep 'project\|repo' |  tr '\n' ' '`
        Size=`echo $line | awk '{print $1}' `
        echo $Size $Rep
done

How can I run it thought Execute shell step in a Jenkins freestyle job?

I need also to add an ssh command to the environment. This should be executed using passwordless authentication.

Note: I don't want to connect to the environmnet and run this shell, but want to execute it directly from the box.

You can specify the script in the "execute shell" step of the build. Where it gets executed will depend on what labels the job needs, and which nodes those labels are associated with.

It sounds like you want to execute this script on an agent. As long as the agent is available, you can request it via a label in the job configuration. It's a bit difficult to determine this without having access to your setup, but if e.g. you have a set of agents on Debian boxes, you could attach the label debian to them, and specify that the job should run on debian nodes in the job configuration.

If on the other hand it should be run on master (where Jenkins itself is running), you can specify the label master.

You thus need to put your script into the "Execute Shell" box that Jenkins provides you, perhaps declaring what shell and options you want to use:

/bin/bash -xe
du -sh /bbhome/shared/data/repositories/* |sort -h |tail -20 |
while IFS= read -r line;do
        DIR=`echo $line | awk '{print$2}'`
        Rep=`cat $DIR/repository-config |grep 'project\|repo' |  tr '\n' ' '`
        Size=`echo $line | awk '{print $1}' `
        echo $Size $Rep
done

  I need also to add an ssh command to the environment. This should be executed using passwordless authentication.

For this to happen, you need ssh keys to be exchanged with the host onto which you want to ssh into.


When does Jenkins clean up old builds? I have existing Jenkins (2.90) jobs that are taking up disk space. When I enable the "Discard Old Builds" option I don't see any cleanup going on in the folder on the builds older than 7 days.

When does Jenkins do this cleanup? Or does it only apply for new builds after I apply the setting?

Per the Jenkins documentation:

  Cleanup per project/job is performed after that job runs.

  As of v2.221, cleanup will also periodically run using the new Global Build Discarders (even if a job has not recently been run).

Source.


How do I specify a database name for a SQL DB Deploy task? Using Azure DevOps, I'm trying to setup a SQL DB Deploy task as a part of a new release pipeline. Database Name is a required field for this task. However, the field is read only, so I cannot manually input it.

There is a tooltip stating that "This setting is linked to the 'Database name' (Parameters.DatabaseName) pipeline parameter." I'm not sure what this means, but it is perhaps the key to my solution.

I tried adding a Pipeline variable named DatabaseName and also Parameters.DatabaseName, but this did not help.

How can I specify a value for the Database Name property?

It seems like Database Name is linked as a Process Parameter.

Under the "SQL Deployment" process, you should see the variable there in an editable state. You can either fill in the value there, or you can unlink and fill in the value in the "SQL DB Deploy" task.


Why is build size such a concern? I often hear (from people, but also from informative CLIs) that the "build/slug size is large". This is particularly so when the build is 0.5 - 2 GB in size

Why (or under what circumstances) is build size such a concern?

Note: the reason I ask is because I see resources like storage and compute as being relatively cheap compared to in the past, so, if anything, I would expect build size to be less of an issue now than in the past

When I raise build size issue as a concern, it usually doesn't come from "it is so big, it will be expensive to store it".

The main problems with large builds are the following -

increased shipping time. moving big bits from place to place frequently is time consuming.
frequent changes to big artifacts plus a large enoug retention period make storing such artifacts become costly, less so with layered artifacts like docker images.
creating such big artifacts usually is time consuming, more so than creating much smaller artifacts. automating the process to create smaller artifacts might take time, but the automation that is repeatable should be as short as possible to allow for fast feedback.
recovering from failure (depending on configuration) might take more time with larger artifacts, especially when an older artifact needs to be reapplied instead of a faulty new-er one.

I subscribe to the four devops metrics:

Lead time for changes - shorten it
Deployment frequency - increase the frequency
Time to restore service - shorten it
Change failure rate - reduce it to never

Large artifacts usually create a problem in each of these metrics, and none of these metrics are really concerned with the cost of storage really - because that is cheap, time is expensive.


Is it possible to create a module that either creates a resource or sources a data source, based on a boolean value? Basically, what I'd like to do is create a module that does this pseudocode:

if <boolean> then
  resource "<resource type>" "name" {
    # ... set variables ...
  } else {
  data "<data source>" "name" {
    # grab data
  }

output "resource_type.name" {
  if <boolean> then
    value = resource_type.name
  else
    value = data_source.name
}

I hope I'm being clear. Happy to add more info to clarify what I'm hoping is possible.

You can use the count meta-parameter to achieve the effects of an if-else statement.  It is talked about in detail in the link below and I also pulled one relevant example out.

I'm not 100% sure if you can give both the data source and resource the same name without a conflict; but I assume it would work.  It may be dependent on the resource you're creating though, not sure.

https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9

Excerpt:

# This is just pseudo code. It won't actually work in Terraform.
if var.give_neo_cloudwatch_full_access {
  resource "aws_iam_user_policy_attachment" "neo_cloudwatch_full" {
    user       = aws_iam_user.example[0].name
    policy_arn = aws_iam_policy.cloudwatch_full_access.arn
  }
} else {
  resource "aws_iam_user_policy_attachment" "neo_cloudwatch_read" {
    user       = aws_iam_user.example[0].name
    policy_arn = aws_iam_policy.cloudwatch_read_only.arn
  }
}

To do this in Terraform, you can use the count parameter and a conditional expression on each of the resources:

resource "aws_iam_user_policy_attachment" "neo_cloudwatch_full" {
  count = var.give_neo_cloudwatch_full_access ? 1 : 0
  user       = aws_iam_user.example[0].name
  policy_arn = aws_iam_policy.cloudwatch_full_access.arn
}
resource "aws_iam_user_policy_attachment" "neo_cloudwatch_read" {
  count = var.give_neo_cloudwatch_full_access ? 0 : 1
  user       = aws_iam_user.example[0].name
  policy_arn = aws_iam_policy.cloudwatch_read_only.arn
}


Using helm to manage an "environment?" I'm new to Kubernetes, and hence helm, so I've been reading a lot, and teaching myself by setting up a cluster, services, deployments, and pods. In my case I use AWS EKS.
I now at least understand each piece from a technical point of view. But now I want to put it all together in managing an "environment." I believe this is the idea behind GitOps.
Say I'm a QA person and am testing a "staging" environment. This environment comprises a number of microservices with their own respective versions. So logically I can represent it this way
/staging-2021-03-06-4 # 4th build of staging environment on 2/06/2021
- microservice1:1.1.0
- microservice2:1.2.0
- microservice3:1.3.0

When we build the next staging, e.g, staging-2021-03-06-5, anyone of those microservices versions can change.
But now I, the QA guy, find an issue with above staging-2021-03-06-4 build, and I ask a developer to look at it. I want to be able to just tell the developer to "check out" the staging-2021-03-06-4 environment so he/she can have exactly the environment that I have to reproduce the issue.
I'm thinking I can set up this idea of "environment" with helm, since the "environment" is nothing but a package of packages? Is this the correct utility of helm? If so, do I just set up a staging-2021-03-06-4 helm chart that has dependencies of other charts, and those other charts are for each microservice?
I'm also reading How to manage 10+ team environments with Helm and Kubernetes? since it applies to me also, but that describes a different (additional) problem I'm trying to solve with helm.
Update: I think I'm essentially wanting GitOps, so I'll read up on that.

Helm charts should not be built for a specific environment, IMHO. You can see that many of the official helm charts have a set of default values and they are configurable from outside. You also can create a helm chart like that and declare your deployment + environment-specific values with another tool like Helm Operator or ArgoCD. In this way, you can both simplify your build pipeline and increase dev/prod parity. You can use that one chart and promote the same build into different environments so you can be sure that different environments are identical except the configuration.
Example ArgoCD application manifests:
production-application.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-yourservice
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourorganisation/helm-charts
    path: yourservice
    targetRevision: master
    helm:
      values: |
        environment: production
        replicas: 10
  destination:
    server: https://kubernetes.default.svc
    namespace: production-yourservice

staging-application.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-yourservice
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourorganisation/helm-charts
    path: yourservice
    targetRevision: master
    helm:
      values: |
        environment: staging
        replicas: 1
  destination:
    server: https://kubernetes.default.svc
    namespace: staging-yourservice


Is containerized deployment of jenkins on ECS better than running jenkins inside EC2 instance to secure pipeline information? I am new to AWS, docker and jenkins. I want to run a jenkins build pipeline and I have tried using following two ways on AWS:

Running jenkins inside the EC2 instance following the steps at this AWS link. I created a new EC2 instance and installed jenkins inside the ec2 instance (ssh into the instance). I configured jenkins and ran a build pipeline successfully.
Running jenkins inside the ECS cluster following the steps at this AWS link.  I created two container instances for Jenkins and data volume, jenkins is installed and running inside one of the instances. (I could not move ahead this point because I was unable to not find the initial password for jenkins)

I could not find much information about the second process of containerized deployment on internet. As I am moving ahead in this process, I am unable to understand which process is better to run a build pipeline securely.

Running Jenkins on EC2 is the better option. It's better to keep the functions that manage your containers separate from the containers themselves. I know this is different than Kubernetes, but this isn't Kubernetes.

Build an ELK Stack on EC2 as well and send your build logs to that instance so you can go through them when troubleshooting. It sounds like your setup is good with data volumes and such. I would recommend that you use Fargate instead of EC2 nodes for your workers though. Not clear which you are using.

There are simpler solutions than Jenkins as well. You may want to consider CircleCI, GitLab Runner, TeamCity, Bamboo, and others. Jenkins is made to do all things and is a good place to learn the process, but if an alternative does the job just as well and more easily, then try that. My current favorite is GitLab Runner. It's just simpler and it integrates your Git repo with the CI tool in addition to having its own image repo for you to use (in certain versions).

You can also give EKS a shot. I haven't tried it but Kubernetes is a superior product. It's far more complex, but definitely a better orchestration tool.


Securing communication between cloud VMs without private networks. Any strategy besides building own VPN? Spiped? I am working on an application that requires a number of different services (web application + database + worker queues, XMPP server for messaging, redis, central authentication via LDAP, etc). For my first attempt at designing the thing, I am estimating that I will need ~15 instances of varying sizes: more RAM/SSD for the database, less so for web server and LDAP, more cores for the worker queues, and so on.

I don't want to use the big cloud Providers (AWS, Azure, GCP) for matters of cost. This is something I am still bootstrapping and GCP pricing calculator put this at ~$5800/month. A similar (in terms of CPU/RAM/SSD) spec in Hetzner cloud would be ~$430/month.

Of course, the difference in price also comes with some difference in features. Hetzner Cloud VM instances do not have any sort of private networking. So I am wondering how I can ensure that all those machines can be effectively protected from the public internet. Some of these services (web server, xmpp server) should also have a public IP, but ideally all other services should have all ports closed (including SSH)

I know that one of the answers is "build your own OpenVPN AS". However, (a) I never done it before and (b) I am under the impression this would mean yet more services to add to this infrastructure.

One thing I did do before was to connect machines via spiped. This does not give you a private network but at least ensures that only those with the key can connect to your public-facing machines.

So my question is: am I missing some alternative to these two above?

Also: docker. I have limited experience with Docker and Kubernetes, but if I went this route it seems I would end up going to the arms of the big providers, not to mention it would still be yet-another thing to learn and take time from my application development. Is Docker Swarm something that could solve this requirement?

I am going to suggest an approach that I would not necessarily recommend, because I do not consider it a manageable, scalable solution:

SSH port forwarding.

SSH is a Swiss Army knife in terms of its features. Virtually any port can be securely forwarded to any other port on a remote machine, and indeed through a remote machine to a target machine beyond.  Documentation abounds, but I really like the drawings in this answer. I use ssh across bastion hosts regularly in a manner that mirrors the diagrams at the link.

As I said, I don't think this is a good solution, but it might serve to give you some ideas and it could be one component of a larger solution.

Good luck!


How do I upload my zipped lambda code into AWS Lambda? So I've read through the docs at: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

I've created the zip file, how / where do I actually upload / publish this zip file into AWS?

It's kind of hidden away and not very intuitive.

Within the AWS Lamba admin: Create Function > Author from Scratch.
Within the Code entry type dropdown there is a Upload a .ZIP file option.


Why do I need tools like minikube or microk8s to run k8s locally? I'm just getting started on k8s and got stuck on running k8s locally (it's running, but I don't understand exactly)

Pretty much all the tutorials on running k8s locally use minikube and I'm a little bit puzzled.

Isn't docker + k8s combination enough to run k8s locally?

Why do I need an extra layer with a virtual machine in order run k8s locally?

I've read about minikube and followed tutorials on the offical k8s website, but I'm still puzzled.

Can anyone shed light on me?

Kubernetes has several system-level parts.  It needs a functioning etcd cluster, the API server proper, and a per-node service called the kubelet.  It also generally expects to be able to use 100% of the resources available on any given node for its own use.  Just having the kubectl binary on its own isn't enough to use Kubernetes; you also need the rest of the cluster that goes with it.

There's no specific reason you can't directly install all of the Kubernetes cluster parts on the host (using kubeadm, or Minikube's "none" VM driver), but it will be trickier to maintain.  It may or may not conflict with local Docker use, and it will be tricky to uninstall or upgrade it.  If you run Minikube in a VM (or Kind in a Docker container) then all of the parts are preinstalled for you, and you can easily stop or delete the VM (or container) when you don't need it any more.