Jenkins pipeline pass upstream build number to downstream job using Pipeline script How to pass upstream build number to downstream job in Pipeline job using (Trigger parameter build on other projects) ?

Agenda: To pass upstream job build number to downstream job to download upstream artefacts in downstream, here's my Jenkins pipeline script

Upstream job script:

node
{
    stage('Build parameters')
    {
    properties([parameters([string(defaultValue: '***', description: '', name: 'version', trim: false)])])
    }/*
    echo env.BUILD_NUMBER
    def Latest_Build_Number=env.BUILD_NUMBER
    build job: 'Downstream', parameters: [[$class: 'StringParameterValue', name: 'Latest_Build_Number', value: 'env.BUILD_NUMBER', propagate:false]]
}

**Downstream job script**:
node
{
    stage('Downstream')
    {
    properties([[$class: 'ScannerJobProperty', doNotScan: false], parameters([string(defaultValue: '', description: '', name: 'Latest_Build_Number', trim: false)])])
echo Latest_Build_Number
}
}

Error:
*[Pipeline] {
[Pipeline] properties
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.UnsupportedOperationException: no known implementation of class hudson.model.JobProperty is named ScannerJobProperty
    at org.jenkinsci.plugins.structs.describable.DescribableModel.resolveClass(DescribableModel.java:478)
    at org.jenkinsci.plugins.structs.describable.DescribableModel.coerce(DescribableModel.java:402)
    at org.jenkinsci.plugins.structs.describable.DescribableModel.coerceList(DescribableModel.java:514)
    at org.jenkinsci.plugins.structs.describable.DescribableModel.coerce(DescribableModel.java:388)
    at org.jenkinsci.plugins.structs.describable.DescribableModel.buildArguments(DescribableModel.java:341)
    at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:282)
    at org.jenkinsci.plugins.workflow.steps.StepDescriptor.newInstance(StepDescriptor.java:201)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:264)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:178)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
    at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)*

Remove the quotes around env.BUILD_NUMBER.

Instead of:

value: 'env.BUILD_NUMBER'

Try:

value: env.BUILD_NUMBER


Declarative Jenkins Pipeline to Poll different branches? We have a gitlab enterpise that host multiple project in one repo. Something like:

Repo/
  projectA/
  projectB/
  projectC/

We have three pipelines that will run base on changes(whitelist region) when polling. They will start polling when received a curl via the notifyCommit? in another job.

However, the pipeline can only work on branch that is run previously.

I have also try to use parameters $branch on branch to build and update the curl job with parameters like:

http://ourjenkins/git/notifyCommit?url=git_url.git&branch=$GIT_BRANCH

but, it still will only poll the last run branch.

Does anyone know how we can have the declarative pipeline to poll multiple branches and run only base on whitelist directory changes?

One could use the PollSCM plugin.

https://jenkins.io/doc/book/pipeline/syntax/

triggers { pollSCM('H 4/* 0 0 1-5') }

  The pollSCM trigger is only available in Jenkins 2.22 or later.


Why should a developer care about Docker? Generally a developer cares about satisfying business requirements. He / she might have the expertise in a particular stack or a framework. But should he / she make an effort to learn docker and it's various deployment methods (swarm, kube, mesos, etc) ?

Simply put why should a developer care about docker ?

PS : The parent question to this post is Implication of introducing docker to the development team

Probably not the answer that you're looking for, but an answer nonetheless :)

Learning about docker and its deployment methods could actually be included in the business requirements by making it part of the project or team development environment, just like the code language(s), version control system, compilers, test infrastructure, etc - to work in that team or on that project one needs to know about and use all of these, can't "bring your own" (in most cases).

Things get a bit more complicated if by "a developer" you actually mean the majority of or even entire development team. Pushing a tool in the development environment without actually having any of the developers supporting it will be really tough. Spend the time to create one such supporter first from the team's technical leadership.

Side note: it might also not be necessary for each and every developer in the team to become a docker expert. Pre-established usage recipes wrapped in simple, cheatsheet-ready commands often allow developers to use docker-based solutions without actually knowing too much about their inner workings, which could be fairly acceptable, especially in large teams. Just like being able to contribute code without knowing all the details about how the end-product is being built.


Help! No development infrastructure in company! I've been tasked to implement it! Background:
I've recently started at a company and I've bee tasked to essentially modernize the development team/dept. Currently the problems are:

Developers currently develop on a live site.
Sites are backed up but they have no source control.
No staging process (dev, QA, UAT, live).
No automated build process.
No SDLC.
Minimal (if any) documentation.
Sites use different technologies.
Timelines are insane because of huge development delays.
Development team aren't aware of common tools that can assist with debugging/development.

The plan
My thoughts so far are (in no specific order):

Bring all projects up to using the same technologies.
Throw all active projects into source control.
Implement staging so that we have Dev, QA, UAT and Live.
Implement CI/CD
Introduce tooling and setup training for development team.
Implement coding standards.
Introduce a methodology more suited to how the team currently operates, most likely Agile or Scrum and the tools and documentation templates that go with it.

The Questions

Have you been in this scenario before? How did you address and overcome these obstacles? How long did it take to implement? (as a rough idea)
How would you prioritize the tasks listed above?
What would you recommend I look into/research? (I'm particularly interested in terms of CI/CD and just how to negotiate such a huge task)

I feel like I need a kind of Maslow's hierarchy of needs to get more of an idea of where/how to start and what to build up to.

Closing notes
Any resources, links, just a pooling of materials, etc. would be greatly appreciated.

Sorry - I know this could be considered a vague question and would give an opinionated response but that would provide the most benefit to me since if I knew exactly the right questions to ask I wouldn't need help because I could use a search engine.

Have you been in this scenario before?

Yes. Pretty standard scenario in my experience.

  How did you address and overcome these obstacles?

In very small steps.

  How long did it take to implement? (as a rough idea)

Very long; in one rather large extreme case: over 10 years, and still ongoing.

  How would you prioritize the tasks listed above?

My priority, with additional comments:

Throw all active projects into source control.

This should really be a no-brainer and low-hanging fruit.

Implement coding standards.

Sensibly.

Introduce a methodology more suited to how the team currently operates, most likely Agile or Scrum and the tools and documentation templates that go with it.

If there is no methodology at all right now, then I'd start with a very lenient Kanban first - which basically just means throw up a ticket system and keep track of everything relentlessly. I would hold off with full-fledged Scrum until everything is on track and you're established as a Good Guy.

Implement staging so that we have Dev, QA, UAT and Live.

With this goes more separation; i.e. devs need to be weaned off of the production systems.

Introduce tooling and setup training for development team.

Try to have the teams develop their own tools if at all possible.

Implement CI/CD
Bring all projects up to using the same technologies.

Might just not be feasible, if by technologies you mean languages and libraries. If you mean Docker or OpenShift or whatever, then sure.

The most important tip for you from my experience: avoid, at all cost, to do everything yourself. Give "impulses" to the teams, but they have to do it (whatever "it" is) themselves. If you do all the work (either personally, or with your separate "DevOps team"), then you will, at the end of the day, not gain that much as everybody will just depend on you constantly.


How is Continuous Integration different from using git to integrate external code? I have a conceptual doubt about CI using Jenkins. I know Jenkins can help me to integrate all code built by developers.

So the question is: How is this not already resolved with Github (or any other git tool)?

I can run git stash, git pull origin master, and/or git stash apply to integrate other code with my code.

So I think I am misunderstanding something about continuous integration. Can you clear my mind?

Merging code changes is definitely handled by Git and its tools.  But Continuous Integration also includes testing and building the software. The development teams are provided with constant feedback on the status of their branch(es) and whether they are failing tests, builds, etc.

Jenkins provides the tools to enable this feedback.  By creating a CI "pipeline" in Jenkins the developer gain much better insight into the build, beyond what simple Git merges and stashes provide.


Connecting GitHub to Jenkins running on docker toolbox Started to learn DevOps 3 days ago, so I have a lot to learn!

I have a jenkins running on docker toolbox container and I can access it via 192.168.99.100:8080.
I configured a job and managed to use "Build Now" to get the files from my GitHub repository, the git is public.

I want my job to build automatically after a commit is made in my GitHub but it does not work.

I configured a webhook in my GitHub settings but after I commit, the "Webhooks" page display an error "Last delivery was not successful. Couldn't connect to server."
In the URL inside the webhook I configured "http://192.168.99.100:8080/jenkins-name/"
Is it wrong to use that URL? I am not sure about it.

In jenkins, I am using the GitHub plugin and I configured in "Configure System"  a GitHub Server with my credentials and the "Test connection" button was verified.
In the job, configured my GitHub repository in the "Source Code Management" section and in the "Build Triggers" section I checked "GitHub hook trigger for GITScm polling".

I read it have something with port forwarding..
Will appreciate any help!

Your configuration is wrong. Here you add http://192.168.99.100:8080/ is a local Jenkins URL. Github does not connect with the local URL. Also you give the wrong webhook-URL.

Please add your public IP of Jenkins in your webhook like http://35.24.128.75:8080/" And GitHub webhook payload url is like http://35.24.128.75:8080/github-webhook"

for Full webhook configuration please watch this video: https://www.youtube.com/watch?v=wsnq9uGAWkM&t=460s


What part of the Bamboo pipeline should generate a Docker image? We are using Bamboo to build and deploy a number of Java web applications. Right now, the Build Plans generally produce a WAR file, which is deployed by the Deploy Plans. We are moving our hosting (and building) to docker, and I'm unclear on the best way to handle this in Bamboo. Specifically, I'm unsure of whether the Build Plan or Deploy Plan should be responsible for generating a docker image.
Option 1

The Bamboo build plan does a multi-stage Docker build to tag and publish a docker image (~ 300MB)
The Bamboo deploy plan is responsible for spinning up a container for that image.

Option 2

The Bamboo build plan does a Docker build, and attaches the artifact (WAR file ~ 80 MB)
The Bamboo deploy plan is responsible for creating an image, inserting the WAR file, and spinning up a container

Just wondering if there is a standard or best practice pattern here. Or am I thinking about this wrong? Are there other options I should be considering?

I believe both options are viable routes, but my preference is for Option 1 as I like having the "deploy plan" to be only concerned with starting a container using the built image.


Validating custom Jenkinsfile Currently we're using Jenkins shared libraries which we're planning to use with custom Jenkinsfiles.
Idea here is that developers should be able to manage those custom Jenkinsfiles without any additional help from DevOps team.

In order to implement this feature I need some way how to validate this custom Jenkinsfile.

Even if I have running Jenkins instance with configured shared libraries post to /pipeline-model-converter/validate wont lint my Jenkinsfile correctly as it wont load my custom libraries.

Is there a convenient way how to check custom Jenkinsfile before actually executing it on running Jenkins instance? e.g., checking if method names in Jenkinsfile are correctly written.

As far as I see, there are two possible approaches, either by misusing reusing the Blue Ocean plugin or using the Groovy-based Jenkinsfile unit test tool.

The groovy approach might be a bit of overhead, since it is not a client-side Javascript verification. An idea would be to run an async task, server side that tests the code the developer provides and returns the results to the client. You mentioned that the shared libraries are not being parsed properly. This might be resolvable through the use of mock Jenkins commands and objects. This might also avoid the security risk which is created through shared libraries, which requires users having push rights.

The client-side solution might be to provide a playground via the Blue Ocean plugin or directly present the Jenkinsfile that has been created via a redirect of the Jenkinsfile to Blue Ocean.

All in all, there is no straight forward JS library solution to your problem, as far as I can tell.


Why is IF failing with satl state file from Jenkins? I invoke a saltstack state file "init.sls" from a Jenkins pipeline as below. If job == Yes, it is expected to execute test_command and test_final, else if job == No, it must execute test2_command and test_final

{% if {{ job }} == 'Yes' %}
test_command:
  cmd.run:
    - name: echo "TEST COMMAND EXEC"
{% elif {{ job }} == 'No' %}
test2_command:
  cmd.run:
    - name: echo "TEST2 COMMAND EXEC"
{% endif %}

test_final:
  cmd.run:
    - name: echo "FINAL SUCCESS"

However it is not successful. Is there a better way to achieve using salt state execution?

The jinja syntax error is caused by the {{ around your job var.
Those curly braces are not required as you are already inside jinja block, noted by the {%.

So to get your code working just replace  your {{ job }} by job.
This should be working.


Can GitHub's Pull Request model be used for implementing post approvals? The accepted answer to "What are possible implementations (or examples) of the four-eyes principle?" suggests that GitHub's Pull Request model is a possible implementation.

And my own answer to "How to implement the four-eyes principle for emergency fixes?" explains the concept of post-approvals.

My question: Can GitHub's Pull Request model be used for implementing post approvals also? If so: how? If not, is there anything else in Git (GitHub?) that I could use for it?

Yes, I do believe so.  To explain that I need to lay some groundwork on how I have implemented something similar, I have simplified the model in an attempt to make it as clear as possible.

Assumptions

I am making the assumption here that Jenkins, TeamCity or similar is being used as the CI/CD tool of choice. Additionally GitHub is being used and that there is a well-defined and appropriately controlled branching structure:

Configuration

In this example, GitHub is configured as follows:

The Black 'Master' branch can only be merged into using Pull Requests, direct commits are disallowed.
The Blue 'Development' branch can accept direct commits or merges; in practice, there may be additional restrictions on this branch.
The Red 'Hotfix' branch can accept direct commits and merges.
Required Status Checks are enabled in strict mode to prevent pull requests merging when the branch is failing the build.
If the Hotfix branch is ahead of Master then pull requests into Master will be blocked, either with the Status API or the Pre-Receive Hooks on GitHub Enterprise.

The CI/CD tools are configured as follows:

Builds from the Development branch cannot be deployed to production.
Builds from Master can be deployed to all environments.
Builds from Hotfix can be deployed to all environments.
Deployments from Hotfix will notify some non-development function, for example, the Change/Release team and ask them to perform the post-approval.

Notes

Master is protected as it represents the current state of production, to do this practically you may have another "Release" branch that deployments are made from and only when successful merge into the Master branch.

Key Points

The Blue Development branch is basically a free-for-all.  Hotfix is kind of a free-for-all but any deployments trigger a type of Break Glass by notifying a non-development function who will perform the post-approval and in the process will merge the change into Master.

It's essential merges into Master stop while Hotfix is ahead of Master to:

Prevent a deployment from Master overwriting the Hotfix, which could result in an regression.
Prevent a Hotfix sitting in the Hotfix branch languishing without being merged in.

In some organisations it may help to prevent all pushes to the central GitHub repository while a Hotfix is pending post-approval.


What is the difference between SRE and DevOps? What is the difference between SRE and DevOps?

Site Reliability Engineering and Development Operations seem to overlap a lot in detail. How do I know which group is responsible for what, and how do I know what jobs would be appropriate for my skillset?

It seems like SRE is about maintaining servers and network, and DevOps is about maintaining code, is that correct? Isn't there still a fair amount of overlap between those two?

DevOps is about maintaining code, is that correct?

DevOps is not "just" about code, or systems, or any one thing.  DevOps is a very general term that covers all things related to software delivery.

Site Reliability Engineering is a term popularized by Google. From this article https://landing.google.com/sre/interview/ben-treynor.html we can distill their TL;DR:

  Fundamentally, it’s what happens when you ask a software engineer to
  design an operations function.

Operations, Engineering, and Software Development are blurring together.  The degree of automation required to create and maintain a mature infrastructure requires skills from all three.  SREs are admins, and engineers, and developers.

See also:
http://shop.oreilly.com/product/0636920041528.do


How do you monitor varnish cache inside a docker container? How do you monitor varnish cache inside a docker container?

I have found many tools on Google relating to this but we are focusing only on nagios and new relic.

Varnish is monitored with what they call VSM. It's a file in your container. If you bind mount the folder where the files is, you can share the same folder / files in other containers that will be able to read it, and thus will be able to monitor your varnish instance.
See the note about VSM and containers in varnish docs.
I have been able to launch a varnishstat with statistics in a container with varnish in the image, but no varnish running, like this:

docker run --rm -it \
       --entrypoint=/bin/sh \
       -v /var/lib/docker/overlay2/$hash/merged/usr/local/var/varnish/$hostname:/varnish \
      emgag/varnish:5.1.3 varnishstat -N /varnish/_.vsm


Jenkins does not seem to be running inside a container I am getting an error while running the Jenkins file tp build and push the docker image into docker hub .

you can reveiw my code github : https://github.com/iakshaykr/webapp.git (using jenkins_pipe branch not master branch )

Jenkinsfile :

pipeline {
    environment {
        registry= "iakshaykr/webapp"
        registryCredential= 'dockerhub'
    }

    agent { dockerfile true }

    stages {
        stage('Bilding image') {
            steps {
             script {
                 docker.build registry + ":$BUILD_NUMBER"
              }
            }
        }
    }
}

Below solution valid on Error code 127.If you are getting any other error code let me know .

After doing R&D finally got the solution.

1st test your code in Jenkins using shell command which is in build option :

cmd : docker build . -t newimage

2nd test : use the docker plugin (Plugin name :CloudBees Docker Build and Publish)

3rd test : use the build option : pipeline scm

code : https://github.com/iakshaykr/webapp.git
Use : jenkins_pipe branch

Please test the thing ,in case any error please reply back .

Note : First setup/add the credentials for dockerhub in your jenkins for authentication .

If you simply google "jenkins does not seem to be running inside a container", you will see that it's only an informational message. Surely, it could be worded better, but it's pretty much harmless.


Using cached dependencies for pipelines in Azure Devops I want to avoid maven downloading all dependencies from a remote repository in all build

I configured a cache, but all executions still download all dependencies.

azure-pipelines.yml

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

steps:
- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    goals: 'clean package'
    codeCoverageToolOption: JaCoCo
    sonarQubeRunAnalysis: false
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Pipeline.Workspace)'
    publishLocation: 'pipeline'
- task: Cache@2
  inputs:
    key: 'maven | "$(Agent.OS)" | **/pom.xml'
    restoreKeys: |
      maven | "$(Agent.OS)"
      maven
    path: $(MAVEN_CACHE_FOLDER)
  displayName: Cache Maven local repo

- script: mvn install -B -e

I forgot to add a MAVEN_OPTS variable in mavenOptions:
    mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'

My pipeline.yml stayed like this:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

steps:

- task: Cache@2
  inputs:
    key: 'maven | "$(Agent.OS)" | **/pom.xml'
    restoreKeys: |
      maven | "$(Agent.OS)"
      maven
    path: $(MAVEN_CACHE_FOLDER)
  displayName: Cache Maven local repo

- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    goals: 'clean package'
    codeCoverageToolOption: JaCoCo
    sonarQubeRunAnalysis: false
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Pipeline.Workspace)'
    publishLocation: 'pipeline'

- script: mvn install -B -e


Planning a backup system using postgress, chef and vault As the title says I am working on planning a way to backup postgres using chef and vault. Idea is that we already encrypt the backups from postgres using a key (random binary string) and we would like to keep that going. We would like to store that string in Hashicorp Vault. My issue is I am not sure how to get it so that the key is made into a file then a few scripts are run. Then comes the issue that a more senior person wants these backups to only be weekly not every time the cookbook runs which if memory serves me right is every 24 hours.

I am not sure the best way to access the secrets in Vault from a chef cookbook. Nor have it run weekly.

I would have preferred to do this with cron jobs and bash scripts, but sadly that is not an option.

First chef runs as often as you configure it to do.

Second Chef is a configuration management system and not a crontab or an orchestration system.

Use chef to configure cron (that can not be not an option, try to find out why this is not considered) for your backup using consul-template to retrieve the secret from vault. Don't use Chef to run the backups because of the caveats you already mention.

However if you really want to shoot yourself in the foot, just add a guard on your backup command like this (604800 seconds is roughly 7 days):

not_if { (Time.now() - File.mtime("/path/to/your/backup/file.tgz") < 604800 }


why is "docker run" used to connect to redis? Trying to understand what's happening here:
root@mordor:~#
root@mordor:~# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@mordor:~#
root@mordor:~# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
10d38d6f1fa1        bridge              bridge              local
5b5fde2dd7cc        foo                 bridge              local
1a7946d0b10c        host                host                local
0cbbb6d4f486        none                null                local
root@mordor:~#
root@mordor:~# docker run --name rd --network foo -d redis redis-server --appendonly yes
ea782770bcff816a642bc700abd2fadd524e3eeca10820e4c158ae4e72eda139
root@mordor:~#
root@mordor:~# docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
ea782770bcff        redis               "docker-entrypoint.s…"   42 seconds ago      Up 40 seconds       6379/tcp            rd
root@mordor:~#
root@mordor:~# docker run -it --network foo --rm redis redis-cli -h rd
rd:6379>
rd:6379> ping
PONG
rd:6379>
rd:6379> set foo bar
OK
rd:6379>
rd:6379> get foo
"bar"
rd:6379>
rd:6379> keys *
1) "foo"
rd:6379>

I can see how to spin up the community version of redis, I only added the foo network.  But why is the connection with run rather than exec?

In short, docker exec is used to operate an already-running container, while docker run -it starts a new container interactively.
When you ran docker run -it --network foo --rm redis redis-cli -h rd you created a 2nd container, this time using the redis-cli image, which was used to query your running redis container.
Were you to repeat the steps you took, and then open another terminal and run docker container ls, you would see two running containers.
Reference: https://spin.atomicobject.com/2018/10/04/docker-command-line/


Terraform's local state (the 2 tfstate files) contain very sensitive info Part of my IaC that describes my systems for Azure is CosmosDB, and Keyvaults.  Terraform happily created everything and I was able to use an output from CosmosDB (db keys) as an input to the Keyvault (as a keyvault secret) so my db key was safely stored.

Then I poked through my terraform.tfstate and sure enough, ALL the cosmosdb keys (pri and sec, rw and ro) keys are in there plaintext, as is the plaintext of the vaults secrets.

So this leaves somewhat of a quandry here  - These state files are apparently integral and they must be maintained (as far as I can tell) but I really dont like the idea of checking them into any repo.  I guess theres no real good solution here other than locally-maintained repos that are redundant as possible?

Do not store your state in anything besides remote backend. Even if you store it in a remote backend the secrets will appear in a plain text again, this is an ongoing Terraform limitation.

In our case we are using AWS S3 backend ( although we have multi cloud environments ), we've got those key benefits out of the box:

Encryption in transit and at rest.
Access policy configuration.
Durability and availability close to the maximum.
Locking capability.
Versioning.

Please do not go with locally-maintained repos that are redundant as possible.


Kubectl: the server doesn't have a resource type "cronjobs" I ran minikube with:

minikube start --vm-driver=kvm2 --kubernetes-version=v1.8.0

status:

minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.39.93

However, running kubectl get all returns this error:

error: the server doesn't have a resource type "cronjobs"

Anyone have any idea what could be the reason for this?

The answer is appearently that there's a bug in Minikube 0.26. Downgrading the Minikube installation is solving it.


elasticsearch-fluentd helm chart on Openshift 3.11 pod failing I install a fluentd chart on OKD 3.11 with:

helm install kiwigrid/fluentd-elasticsearch \
--name fluentd \
-f ../../tools/fluentd/values.yaml \
--namespace logging

The pod is created, I can enter in it and I can curl to external elasticsearch successfully.
However the liveness probe keeps failing and the pods on timeout get recreated:

Normal   Killing    10m (x3 over 36m) kubelet node1.qa1.isp.ezbob.com
Killing container with id docker://fluentd-fluentd-elasticsearch:Container failed liveness probe.. Container will be killed and recreated.

And I can't find any indexes being created in ES.

The liveness config is the default for the chart:

 Liveness:       exec [/bin/sh -c LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; STUCK_THRESHOLD_SECONDS=${STUCK_THRESHOLD_SECONDS:-900}; if [ ! -e /var/log/fluentd-buffers ]; then
  exit 1;
fi; touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-stuck -print -quit)" ]; then
  rm -rf /var/log/fluentd-buffers;
  exit 1;
fi; touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
  exit 1;
fi;
] delay=600s timeout=1s period=60s #success=1 #failure=3

The OS is CentOS 7.6. I can't get any info about this issue and no trace to catch, it looks like some permission issues on mounting points.

Changing logging driver form journald to json has fixed it.
I edited /etc/sysconfig/docker, and set

OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false'

Then restarted docker on the nodes I needed:

systemctl daemon-reload && systemctl restart docker


Can I persist a Yarn install through different stages in a GitLab pipeline? I was thinking about a pipeline with the following steps:

Install image dependencies (such as java, node, yarn and openssl)
Install node packages needed for the build
Install PMD
Do a code Analysis
Build
Deploy
Run tests on the server
Clean up

But the issue here is that I require the tools installed on steps 1 and 2 for the rest of the build. There's a package that is used to run steps 5 to 7, for example, and OpenSSL is required to decrypt a server key in the repository (so GitLab can connect to the server).

I can't seem to find a way to persist Yarn's install through different stages. When the pipeline moves from stage 1 to stage 2, it will use yarn commands and will fail with the 127 error code (command not found), like this:

$ yarn global add sfdx-cli
/bin/sh: eval: line 81: yarn: not found

ERROR: Job failed: exit code 127

Is it possible to persist the "environment setup" through all the stages? Can artifacts be used for this? If so, will I need to setup the commands for every stage? For example: Yarn might be installed in a folder which will be cached, but I need to point the 'yarn' command to the correct binary, right?

We install things in the before_script section of the .gitlab-ci.yml that way on every run of the runner we know what packages exist for each stage.


Dev-Ops with bitbucket I want to improve my current dev CI workflow, I would like to create a server with ISPConfig and some kind of CI tool like Jenkins

The catch is It has to all sit on 1 server, I see folks saying jenkins needs its own server but im just wondering is that a hard set rule or what.

Or has anyone got any tips on other tools to use which would work better with bitbucket or bitbucket pipelines?

My current setup has been a maze of bash scripts which get triggered when a dev pushes to the master branch but it's not very convenient for updates or adding new sites etc so id love to hear what other people are using to simplify the process of pushing to staging/dev .

I had this problem when I needed to build something at home.

What I did:

installed docker in a Ubuntu server VM
installed docker in a windows server 2016 VM (if you need windows containers like i did)

THEN:
In the Ubuntu machine:

I created a local registry running in a container (1 command :)  )
I spin up a container with a git server - I "linked" the configuration folder to a folder in my machine for persistence.
I spinn a container with jenkins server (same trick with the folder)
I spin up containers with slaves
I have a database container attached to a nexus container for artifacts and build dependencies

In windows VM :

I spin a jenkins slave container to build windows apps
I spin IIS machines to test the built app

My home computer is 32 GB ram / core i7 and it works great , even with windows container databases (these are monstrous)

Tricky for me - work in progress - build a container in a slave running in a container . It can be done but I am on googling phase.

Advantages:

if you use docker compose and you save your containers in the local registry you get to move your work easily . I always version my docker compose files in a github repo.
you can pause the VM host and move everything in some other host.
you can branch your things for having a test env by cloning the machine and copying the host folders


How to pass parameters to a container orchestrated by Kubernetes similar to running the container at the command line using the docker run command? Background

I have a docker container that I run in the following fashion:

docker run --rm my-custom-container:latest --username me --password yes --url "https://my.url"

The container does what what it needs to do with no issues.

I have pushed my container image to a private image repository, and I am able to create a Kubernetes deployment. Kubernetes is able to pull the container from the private image repository for the container with no problems.

Problem

However, when Kubernetes goes to start the container it returns with an error stating that it cannot run the container.

This is due to a lack of parameters being passed to the container to run.

Question

How do I pass parameters to my container during a Kubernetes deployment?

You can include the arguments in the deployment yaml.  The official docs include examples and further customization.

spec:
   containers:
     image: myimage
     name: myContainerName
     args: ["me", "yes", "https://my.url"]


Deploying a pod per REST request? Is it possible to deploy 1 pod per each REST request?  My process starts up in .004s and is about 1GB in size.   After the REST request completes the pod will terminate.

Given that I'd like to deploy the pod/container for this on each REST request. Is it possible to do this in < 200ms?  A sort of serverless architecture using kubernetes.

How would you do this in a reliable / resilient way?

As per my experience, it is not a good idea because the availability of a pod is not directly related to how fast the container is up, a lot of other things affects the time the pod is ready and the time kubernetes sends traffic to that new pod.

You need to think about how your scalability strategy is trying to adapt to your actual traffic and the different health checks change that readiness time.

In Kubernete you have the livenessProbe and radynessProbe used to check two different thinks on the lifecycle of the pod, because of that you need to review the lifecycle of a pod in kubernetes and if your app can adapt to it.

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/

In case you are worried about the number of requests you can handle per second my advice is to have a pool of pods ready to response a good amount of it and take metrics to know better how your app responds to a high demanding traffic context and understand how the loadbalancing of kubenetes work and can affect your app, with that try to scale in a more smart way.


Join a Linux Docker swarm with a local Windows Docker for test purposes I'm currenlty trying to connect/join a Linux Docker Swarm cluster with my Windows machine in order to test a stack without sending the stack over and over to my test cluster under Linux. My problem is that I can't join the cluster with my Windows Docker.

My cluster had initially 1 manager and 4 workers. I recently created and added a new manager to it in order to add my Windows Docker node as manager. With this, I shouldn't have any issue with the manager quorum (since my two managers will be always up, and my Windows Docker could be interrupted at any time).

I configured my local firewall - and also on each nodes of my Linux cluster - to accept connections on ports (the following can be found at the end of this documentation) :

TCP port 2377 for cluster management communications
TCP and UDP port 7946 for communication among nodes
UDP port 4789 for overlay network traffic

The advertising network in my cluster is 192.168.0.0/24, one of my manager's IP address is 192.168.0.100 and my PC's IP address is 192.168.0.143 (same network). They can ping each others and I tested the ports defined before.

So, when I perform the docker swarm join-token manager on a manager I have something like this:

docker swarm join --token SWMTKN-1-27hzpcnek5... 192.168.0.200:2377

If I copy/paste it to my PC:

Error response from daemon: manager stopped: can't initialize raft node: rpc error: code = Unknown desc = could not connect to prospective new cluster member using its advertised address: rpc error: code = DeadlineExceeded desc = context deadline exceeded

But, even with this error, on my cluster, if I run docker node ls, my node appears in the list:

8380zb5jeke0elwohr7aieke8     linuxkit-00155d00d946   Ready               Active

But on my PC Docker thinks that it's not member of a cluster:

Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

The Docker version is different in the cluster and on my PC:

Cluster:

Client: 17.09.1-ce
Server: 17.09.1-ce

PC:

Client: 18.03.1-ce
Server: 18.03.1-ce

Could the issue came from this?

N.B.: I work with Bash on Ubuntu on Windows, but the result is the same on PowerShell.

N.B. 2: I don't know if it's useful, but since Docker for Windows uses Hyper-V, I have a NAT interface: 10.0.75.0/24 (and the VM's IP address is 10.0.75.1). I tried to use this interface as advertising address and listening address but the issue returned is the same

Does anybody already encounter this issue? Any ideas?

Thanks! :)

EDIT 1:

Disaggreeing with @BMitch's answer, after tests, I don't think there is any issue with the difference between managers' Docker versions.

As I said in my previous comment on BMitch's answer, I had to add the option --advertise-addr 192.168.0.200 - which is my leader manager address - to the docker join command. Until this, I used --advertise-addr 192.168.0.143, my PC address.

I hope it will help others.

EDIT 2:

If I run docker info on my PC I got this:

Swarm: active
 NodeID: o9wehjcyhe3n6xxzy1epnhxdp
 Error: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online.
 Is Manager: true
 Node Address: 192.168.0.200
 Manager Addresses:
  192.168.0.200:2377
  192.168.0.200:2377
  192.168.0.201:2377

There are two managers with address 192.168.0.200:2377.

EDIT 3:

I finally gave up and installed a Linux server on another machine in order to join the cluster. That's not what I wanted but I can work now.

If any have information about this issue, I invite you to share it. Thanks :)

Why do you want to create a new Manager node, risking damaging your raft consensus and cie?
I find it way easier to simply expose the docker socket locally, as if you were working inside the node, but with your Windows environment.
To do so, simply open a ssh tunnel that exposes /var/run/docker.sock:
 ssh -M -S ~/.docker.sock \
     -fnNT -4 -L localhost:1337:/var/run/docker.sock \
     USER@MANAGER_IP

Refer to man ssh to see what all these options mean.
It will open an ssh tunnel; you will still be in your local environment shell, only thing left to do is set the proper DOCKER_HOST environment var, so your docker cli is bound to your swarm manager.
export DOCKER_HOST=localhost:1337
And that's it.
Note: Docker also includes an option to do this; I've quickly searched for it but can't manage to find it. It's in the most recent version of docker.


How to build, package and deploy a react-native-web app in a Docker container? Given there is a react-native-web solution, what needs to be done to build it and run inside a Docker container?

What is the build environment/toolchain?
Which environment is required to run?

From what I can find so far, it's not clear how this should work.
UPDATE: Project examples.

It's a pretty general question, I will try to answer the best I can:
If you want to just build and run an image:
Build the project: npx expo-cli init react_native_app --npm

You need to create a Dockerfile file in your repository
Docker build -t {tag_name} . (You can replace '.' eith the Dockerfile path)
Docker run...
If you want you can create docker compose file to better control your app.
Reference:
https://www.rockyourcode.com/how-to-run-react-native-expo-web-in-a-docker-container/


Interacting with a Trident container (NetApp) via CLI within a Openshift/Kubernetes cluster? Background

We're currently using NetApps Trident PV (Persistent Volume) orchestrator to dynamically create volumes on our NetApp ONTAP clusters for Pods running within our Openshift/Kubernetes environment.

NOTE: See links in the references section if you're unfamiliar with Trident + NetApp.

1. Native Built CLI

Most people will use the pre-built binary but this will only run tridentctl on Linux. Previously we'd been using the tridentctl command natively on a Linux system to interact with the Trident controller running within our cluster.

However, I'd like a more native method to do this from macOS. Currently NetApp does not ship a prebuilt tridentctl binary to do this.

2. Using Docker Image

Previously we'd been using the tridentctl command natively on a Linux system to interact with the Trident controller running within our cluster.

Here's my Trident Pod:

$ oc -n trident get pods
NAME                       READY     STATUS    RESTARTS   AGE
trident-6bdbdbc5dd-l8qtr   2/2       Running   14         6d

Remote shell into the Pod:

$ oc -n trident rsh trident-6bdbdbc5dd-l8qtr
Defaulting container name to trident-main.
Use 'oc describe pod/trident-6bdbdbc5dd-l8qtr -n trident' to see all of the containers in this pod.
/ #

Attempt to run tridentctl:

/ # tridentctl get backend
Error: could not find the Kubernetes CLI
/ #

3. Other Methods

I explored my options in vain but did not care for any of these:

Build Trident from source
Construct a new Dockerfile based on the Trident Docker image
Roll my own Docker image with tridentctl + kubectl

References

NetApp Trident Read the Docs
Trident for Kubernetes (overview)
NetApp Trident (Github)

There's apparently 2 methods one can use to leverage tridentctl to interact with a running Trident Pod in their Openshift/Kubernetes cluster.

1. Server string

The tridentctl CLI can be instructed to talk to whatever server you want remotely using the -s or --server argument. In this context you could use the approach of remote shelling into the Trident Pod and then directing it to interact with the local Trident Orchestrator daemon.

For example

$ oc -n trident rsh trident-6bdbdbc5dd-l8qtr
Defaulting container name to trident-main.
Use 'oc describe pod/trident-6bdbdbc5dd-l8qtr -n trident' to see all of the containers in this pod.
/ #

Now to interact:

/ # tridentctl -s 127.0.0.1:8000 get backend
+--------------------------+----------------+--------+---------+
|           NAME           | STORAGE DRIVER | ONLINE | VOLUMES |
+--------------------------+----------------+--------+---------+
| ontapnas_192.168.101.101 | ontap-nas      | true   |     389 |
+--------------------------+----------------+--------+---------+
/ #

2. Creating a Bash Function

To expand on the above method, one could construct a shell function or alias in Bash to streamline this interaction like so:

Openshift

tridentctl () {
    oc rsh -t -n trident -c trident-main \
        $(oc get pods -n trident -l app=trident.netapp.io -o jsonpath="{.items[].metadata.name}") \
        tridentctl -s 127.0.0.1:8000 $*
}

Kubernetes

tridentctl () {
    kubectl exec -t -n trident \
        $(kubectl get pods -n trident -l app=trident.netapp.io -o jsonpath="{.items[].metadata.name}") \
        -c trident-main -- tridentctl -s 127.0.0.1:8000 $*
}

With one of these functions set, you could then run the following command from your macOS system:

$ tridentctl get storageclass -o json
{
  "items": [
    {
      "Config": {
        "version": "1",
        "name": "basic",
        "attributes": {
          "backendType": "ontap-nas"
        },
        "storagePools": null,
        "additionalStoragePools": null
      },
      "storage": {
        "ontapnas_192.168.101.101": [
          "NA_01_aggr1",
          "NA_01_aggr2"
        ]
      }
    }
  ]
}

How it works

The above function merely determines the name of the Pod within the trident namespace and then remotely executes the command tridentctl using the same method above (from method #1) of connecting to the locally running Trident Orchestrator daemon within the Pod using the --server string.

The advantage with one of these approaches is that you simply can use a function/alias to codify it.

Caveats

Keep in mind that with the above shell function/alias approach will not work with commands such as this:

$ tridentctl create backend -f <backend-file>

because the files you'll be referencing with it, -f <backend-file>, will be local to the system where you'll be running the function/alias, but the tridentctl command will actually be executing within a Pod on another system. This Pod will not have access to your local system's filesystem.

Your only options in this situation will be to either copy the file into the Pod's /tmp filesystem and execute it from there or to redirect the contents of the file, <backend-file> through a pipe, (|), for example:

$ cat <backend-file> | tridentctl create backend -f -


ansible error 'first argument must be string or compiled pattern' I have this code in my playbook:

- hosts: standby
  remote_user: root
  tasks:
  - name: replace hostname in config
    replace:
      path: /opt/agentd.conf
      regexp: #\s+Hostname\=
      replace: Hostname={{hname}}
      backup: yes
  - name: add database array in files
    lineinfile:
      path: /opt/zabbix_agent/share/scripts/{{ item }}
      line: 'DBNAME_ARRAY=( {{dbname}} )'
      insertafter: DB2PATH=/home/db2inst1/sqllib/bin/db2
      backup: yes
  with_items:
   - Connections
   - HadrAndLog
   - Memory
   - Regular
 - name: restart service
   shell: /etc/init.d/agent restart
   register: command_output
   become: yes
   become_user: root
   tags: restart
 - debug: msg="{{command_output.stdout_lines}}"
 tags: set_config_st

it will replace # Hostname= in a config file with Hostname= givenhostname and add an array in 4 scripts. array is the name of given database. then it will restart the agent to apply the changes.
when i run this command:

ansible-playbook -i /Ansible/inventory/hostfile /Ansible/provision/nconf.yml --tags set_config_st --extra-vars "hname=fazi dbname=fazidb"

i get this error:

first argument must be string or compiled pattern

i searched a bit but couldn't find the reason. what should i do?

The task which fails is this one:

 - name: replace hostname in config
   replace:
     path: /opt/agentd.conf
     regexp: #\s+Hostname\=
     replace: Hostname={{hname}}
     backup: yes

Try like this:

 - name: replace hostname in config
   replace:
     path: /opt/agentd.conf
     regexp: '#\s+Hostname\=.*$'
     replace: Hostname={{hname}}
     backup: yes

Note that if you don't add .*$, the task will only replace # Hostname= and you will end up with a Hostname=oldhostnamenewhostname.


On-Site Octopus Deploy - How to deploy projects in a multi-project solution to different IIS locations I'm tasked with setting up CI/CD for my company. I'm using TeamCity which I have configured using the Octopus Deploy plugin to create a Nuget package to the appropriate dev server (alternatively, I can have TeamCity spit out a .zip).

The package itself is a solution that contains several projects, and each project needs to be deployed to different folders on IIS. Is this something that I can accomplish with Octopus Deploy, and if so how?

I've looked through the documentation and spent a lot of time Googling but I haven't found an example of someone doing this but I want to use Octopush Deploy because of the reporting and flexibility it provides (since I will need to set up pipelines to other environments in the future).

Information about the solution and environment:
Multi-project solution
.NET Framework 4.6.1
IIS version 6.2
Windows Server 2012 R2
Octopus Deploy v6.0 on-site
TeamCity Professional 2019.1.1 (build 66192)

The solution is currently deployed on ISS like this:

Any assistance provided or links to resources is deeply appreciated.

Kind regards.

It can be achieved, however, you will make your life easier if you modify your TeamCity build process to create a package for each IIS Website then deploy each one individually using a Deploy to IIS step.

If it's impossible to change your build process then you will need to do some pre-processing or just deploy the websites through a custom script.


Running Jenkins in docker with custom uid / gid I have the following Dockerfile:

FROM jenkins/jenkins:latest

I'm build a Jenkins container as follows:

docker build -t my-jenkins --build-arg uid=1003 --build-arg gid=1003 .

I then run the image as follows:

docker run -it -p 8080:8080 my-jenkins

I go to localhost:8080 and finish installation (without any plugins selected).

I then create a job with a single shell command:

id

I run the job and I expect the output of the job to be

+ id
uid=1003(jenkins) gid=1003(jenkins) groups=1003(jenkins)

The output however, is

+ id
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins)

which seems to indicate that my --build-args for uid and gid are completely ignored.

Why is this? The Jenkins Dockerfile should accept uid/gid as guild args.

As usual I think I came up with the answers shortly after spelling out the problem carefully here. I would be great if someone could confirm that my understanding is correct.

TL;DR: It's not possible to use --build-arg for FROM images

The FROM command basically tells docker what binary image to start from. The FROM image is not built recursively. This obviously means that --build-arg won't be taken into account in the FROM image.

So why is uid and gid even ARGs in the first place if I can't use them?

It is possible to use them in a reasonable straight forward way. docker build accepts git repo URLs when building, so you can indeed kick off a Jenkins docker with custom uid and gid quite easily as follows:

docker build -t my-jenkins --build-arg uid=1003 --build-arg gid=1003 https://github.com/jenkinsci/docker.git
docker run -it -p 8080:8080 --rm my-jenkins

But I need to create my own Dockerfile :-(

If you really need to custobmize the Jenkins image and need a custom uid / gid, the best way forward is to change the uid / gid of the Jenkins user in your Dockerfile. The uid = 1000 and gid = 1000 is already hardwired when your part of the Dockerfile kicks in.


Best way to launch 100 identical Docker machines on AWS? I need to do some large scale testing on a system. The system consists of 2 main parts: Manager and Nodes. There will be only one manager server, but 100 nodes. Both the manager and nodes are Docker images.

I want to be able to spin up a setup on AWS, where an instance of a manager, and 100 instances (one instance per server) of the node is started.

What would I use for this? The manager and nodes should be able to communicate together, and the nodes should not be load balanced in any way. So they need a dedicated internal IP that the manager can communicate with.

I've looked at ECS and CloudFormation+EC2 with a custom AMI, but I'm not sure what is the way to go.

Ideally I want a script that I once in a while can run to launch the servers, and close them down easily.

Any suggestions on what I can do on AWS to launch this setup with ease?

Look at AWS Fargate - it lets you run your Docker containers without having to spin up the EC2 instances for an ECS cluster. You simply schedule the Tasks (or Services), each gets is own IP address and they can talk to each other.

You can also use AWS Service Discovery to facilitate the registration and lookup of the tasks' IP addresses.

BTW Note that the default Fargate concurrency limit is 20 Tasks/Services - if you need 100 of them you'll have to raise a Service Limit increase support ticket.

We've got it raised to 200 concurrent tasks because our processing runs 160+ containers in parallel, all on Fargate without a single EC2 to manage.

Update - how to spin up 100 containers

I can tell you how we do it with Fargate - in our CI/CD pipeline we build the container image and upload to ECR. Part of the source is also a CloudFormation (CFN) template and in the Deployment stage of the pipeline we create/update CFN stack which creates ECS TaskDefinition with all the container parameters.

Then we've got a scheduler job written in Python and triggered by CloudWatch Event (cron-like scheduler) that looks up the TaskDefinition ARN and runs 160+ Tasks from that single TaskDef. You can give each Task an extra parameter, or override some config, etc. Or not and just spin it up.

We used to use EC2/ECS and had to do an extra step first - scale out the ECS AutoScaling group to tens of nodes to support our workload and then at the end scale it back in to 0. Other than that it's the same whether you run it on Fargate or on EC2/ECS.

Hope that helps :)


Ways to the DevOps hiring process optimization through CALMS paradigma? Much of DevOps recruitement happens to follow along the lines of keyword matching, which leads in my opinion to solely technology focus.

Now, DevOps is about so much more than just technology, and DevOps Engineer is not just a better system administrator with some coding skills.

Senior DevOps role/profile means to me also offering seniority in many other foundations and practices beyond infrastructure and software engineering skills like Lean, Measurement and being open and communicative (who asks DevOps hires for their communication skills, honestly?!)

So, can a job ad / interview be more efficient in some way - for example, by applying questioning CALMS categories as well? - Leading to questions like "now, how do you apply lean principles? How were cultural aspects addressed in your recent DevOps projects?"

Further elaboration:

Culture (e.g. strategies for conflict management and attitude to failures, own and others')
Automation (here you ask about Puppet/Docker etc skills)
Lean (foundations of Lean? Waste types?)
Measurement (ask for tools like JMeter but go also to things like sampling, data modeling..)
Sharing (obviously knowledge management and according tools)

UPDATE - so why wouldn't employers/recruiters structure the interview by CALMS as shown below (additionally, the "automation" section could be formulated along the DevOps toolchain model (document link, readonly)?

Side note - so culture for example is actually not just a soft skill anymore, for DevOps it is one of the core skills - like all the others in this domain.

This is a brilliant idea, also because of Daniel Kahneman who showed that if you split a single score into 5 weighed scores and add numerical criteria and bounds to those, you will significantly reduce bias. You could design not just the resume scoring, but the entire hiring process, with phone screens, onsite interviews, everything in this way. It would significantly reduce the inherent bias of the interviewers. We have actually started to do something similar for all hiring.

Obviously, inside each area, your should add weight to what is important to the company for the position, but you are hiring a well-rounded engineer and you want someone who will propose major changes to how your organization operates, you are not simply hiring someone for specific skills to work in a limited area. Many people simply see this role as a higher paid Release and Build Engineer and if that is the case, that is what you should hire and advertise for.

For a DevOps hire, I would suggest replacing the Lean with Learning. It is originally CAMS and even though some extend it to CALMS to include Lean, that is somewhat restricting as DevOps is based on so much more than just Lean. It is also Deming's ideas about Special and Common Cause Variation and System Thinking, Nash's Equilibrium (if each optimizes for themselves, the result could be suboptimal, compared to when everyone includes the interest of the group), Shewhart's Statistical Process Control, Goldratt's Theory of Constraints, Taleb's Anti-Fragility and so many more.

This would also allow you to include participation in conferences in Learning and presentations at conferences or meetups as Sharing. In a position where you are not always a part of a team or your company might not be big enough to have your peers as your co-workers, it is even so much more important to establish and maintain out of workplace relationships and learning opportunities. We generally grouped those two under Culture.

I would personally put under Culture the soft skills required to be effective in improving processes at your organization. CMMI, Kanban, Work in Progress limits, Agile practices, etc.

JIRA seems more like Sharing tool and Git is more closely related to Automation.


Are there Docker Hub comparable public Docker image repositories? Historically, you can download official community software from a distributed mirror network.

Even for Java Maven, there are two sites globally.

But what is about Docker Hub? What if Docker, Inc. decides to close the Hub or the site will become unstable?

Beyond just having business critical images which you can rebuild and store locally, Docker Hub is also:

indicator that a release is official
often provides Dockerfile which is very valuable configuration knowledge
provides links to backing GitHub projects
features like starring and download numbers

Are there other mirrors not hosted by Docker either comparable initiatives so that an open source community would say "ok we can publish our release also there"?

After a short research, I see there are many products to setup your own registry, but not an alternative Hub site.

With Docker following the Open Container Initiative, in the event that Docker were to disappear, people would still have access to images stored outside of Docker Hub.

While there is not a public mirror of Docker Hub, you do have access to other registries that often store the official image.

For example, if you were to pull the official Docker SQL Server image, you would target mcr.microsoft.com/mssql/server:2019.  So even though the image can be searched on in Docker Hub, the image still exists outside of Docker Hub.  The three big cloud providers (Azure, AWS, and Google) all have container registries that allow public access.  Many of the cloud registries support storing of images in regions and would allow "mirrored" architectures that are transparent to the public and disaster recovery (i.e. end users only pull from one link and the provider routes to an active available registry).


System test frameworks for scale testing on physical and virtual systems I'm searching for a framework for running system tests. The requirement are:

Able to manage provisioning to multiple physical machine
Able to manage provisioning to multiple virtual environments (AWS, Azure, etc)
Able to run tests on a given machine be it physical or virtual.
Not bound to a single language or platform. We wish to at least test against Linux, Windows at least.

Is there anything that meets those requirements?

Thank you!

Test Kitchen? Your requirements are huge and vague but it's probably your best bet.


Condition CircleCI build on what part of monorepo changed? I'm working on a monorepo project that contains a few different subprojects in different languages. Right now I run three different CircleCI "jobs" on each commit. However, each of these packages compile and test themselves independently. Hence I don't actually have to spend time on running all three jobs every time.

Is there some way to condition a CircleCI job on which directory you committed code to?

This discussion gives a pretty good suggestion on how to accomplish what I want.

https://discuss.circleci.com/t/does-circleci-2-0-work-with-monorepos/10378/11


IaC Deployment in Azure with Terraform + Chef vs. Puppet vs. Ansible for Data Engineers I am an Azure Data Engineer and try to leverage the IaC benefits in this area of Azure Data Factory, Azure Databricks, Azure SQL & Synapse, Azure Power BI, Azure IoT.
After some research I figured out that I need two essential parts:
1.Provisioning part
2.Configuration management part

Breaking down to technology I decided to use Terraform for the provisioning of infrastructure like (RG, Network, DBs etc).
For the configuration management I am still not sure what to choose between Chef, Puppet and Ansible.
There is another idea to use Docker/Packer + Kubernetes.
What are the best practices of IaC and automation when it comes to the Area of Data Engineering/Big Data?

Cloud setup could be done completly by Terraform. Check out terraform provider resources for required parameters ex. Azure Data Factory
Configuration management stuff mostly in use for OS tuning inside Virtual Machines. So maybe you don't need them at all.


How to make a Jenkins API call when using "Log in with Google" as security? Jenkins API calls are generally made using a username/password combination, or other set of credentials.

We have a Jenkins instance using "Log in with Google" authentication, meaning no credentials are stored in Jenkins for users since login happens with OAuth. I can neither make a new user with creds of my choice, nor view my own credentials to use for the call.

I intend to build jobs using the API, so setting the Anonymous roll to be allowed to build isn't an option. If it matters, my goal is to trigger jobs via an AWS Lambda.

How do I get credentials to use for API calls?

Typically you authenticate API calls (I assume you mean HTTP REST API; there is also a Groovy API) using an API token.  The API token can be generated from a user's profile page while logged in as that user (Profile page > Configure > API Token).  More information about generating and using API tokens is available in the official documentation.  There is also a question about API tokens posted to StackOverflow.


Is it possible to combine failure stage from groovy with try catch from pipeline? I have a pipeline with some stages and logic. The only possible debugger is adding try-catch to each stage:

stage ("distribution"){
            steps{
                script{
                    try{
    amd_distribution_distribute_bundle
                    }
                   catch (Exception e) {
                    echo e.toString()
                    }
            }
        }
    }

however, if one stage fails, I need to stop the pipeline:

status = sh script: '''
                set +x
                python3.4 main.py --command create_bundle
                ''',  returnStatus:true
             }
        }
     }

    if (status == 1) {
    error("returning 1 from func")
    }

but the try-catch from the pipeline actually cancel it (it proceed to other stages. is there a way to combine them?

You may use and update a flag to false in the catch and then use that flag in the when directive of other stage(s) that you want to skip because of any failure in the given stage. Example:

boolean flagStageOneSuccess = false

pipeline {
    agent any

    stages {
        stage ('stage-1') {
            steps {
                try {
                    // Do something that might fail

                    flagStageOneSuccess = true

                } catch (Exception e) {
                    // Handle the exception
                    flagStageOneSuccess = false
                }
            }
        }

        stage ('stage-2') {
            when {
                equals expected: true, actual: flagStageOneSuccess
            }
            steps {
                // Do something in stage-2
            }
        }
    }
}

It would also benefit you in viewing the status of your stage in Blue Ocean. If the stage is skipped, it shows as 'Not built'.


How to measure Lead Time for a Scrum Team I'm currently revisiting Accelerate by Nicole Forsgren and I'm looking at her metrics for measuring high performance. Specifically Lead Time which she defines as:

  Lead Time is the time it takes from the customer making a request to the request being satisfied.

She also notes that

  there are two parts to the lead time: the time it takes to design and validate a product or feature, and the time to deliver the feature to customers.

However, in a normal scrum team stories are added and prioritised to the backlog all the time. Something may be added and then deprioritized or even cancelled. when exactly does a customer "make a request?"

Is it:

When it's added to the backlog?
When it's accepted in Sprint planning?
When it's "accepted" and estimated?
Somewhere in between?

I'm less worried about "delivered" that feeds into definition of done.

Any "word of god" statements from Forsgren herself would be preferred.

The idea of lead time is that when the customer asks for something, a clock starts ticking in their head and that clock stops when they have it in their hands. Therefor, a feature's lead time would be from when the request is received (this usually corresponds to adding it to the backlog) until it is delivered and usable to the requestor.

Full disclosure, I haven't read Accelerate, but since Forsgren is using an existing term and it seems it's being used correctly, I am referring to the proper definition of the term.

Now, that isn't to say all those other things you mention aren't valuable. They may be. They may even be more valuable to you than lead time. They just aren't lead time.


Ansible: Test if multiple RDS instances are all stopped With Ansible, I'm stopping several RDS (it could be EC2) instances.

Then I want continue only when all of them are in the state stopped. I use the following task:

- name: Wait for RDS modify to finish
  rds_instance_facts:
    aws_access_key: "{{ assumed_role.sts_creds.access_key }}"
    aws_secret_key: "{{ assumed_role.sts_creds.secret_key }}"
    security_token: "{{ assumed_role.sts_creds.session_token }}"
    db_instance_identifier: "{{ item }}"
    region: "{{ region }}"
  loop: "{{ vars[ 'list_rds_resources_ids1_' ] + vars[ 'list_rds_resources_ids2_' ] }}"
  register: aws_rds_facts
  until: aws_rds_facts[0].db_instance_status == "stopped"
  retries: 90
  delay: 10

Right now, my code is testing only the first element. How to make the test on all existing elements at the same time ?

If I understood you correctly, you need free strategy.
It will run all tasks ASAP, not waiting for anything.
https://docs.ansible.com/ansible/latest/plugins/strategy/free.html

Bear in mind, that logs will be a mess :)


Is there a good standard to follow when choosing a service mesh for your microservice application? The industry appears to have chosen the de facto standards for containers (docker) and orchestration (kubernetes) but this doesn't appear to be the case for the options for implementing a service mesh.

Are there some good standards to follow when choosing from the available options for implementing a service mesh?

I would say that Istio is the de facto standard for Service Mesh. It is launched by both Google and IBM.

Welcome to the service mesh era: Introducing a new Istio blog post series
IBM, Google Cloud and the open community launch Istio 1.0 to bring microservices to the enterprise


What means “system engineer practicing DevOps”? In the comments section of this answer, it’s explained that the asker is a “(system) engineer practicing DevOps” not a DevOps engineer, and this it is an “important distinction” not often understood.

So, what makes a DevOps Engineer NOT a DevOp Engineer? What is the distinction between a System Engineer and DevOps Engineer that is not often understood?

What the comment means is that DevOps is a way of working, a culture if you will, not a job title. There are plenty of blog posts about it.

While they have a point, one could also argue that Cubism is a way of painting, but "cubist painter" is still perfectly valid.

This is a potentially long and opinionated discussion, which is out of scope of this forum.

The point is, the definition of DevOps is too lacking on specifics about what exactly the person practicing it does and what kind of tools they use to do it to be of any actual use as a job title. Which is why it is often misunderstood and misused. And there is (almost?) always a better title to describe the job.

For example, if your job consists of solely setting up pipelines in Jenkins, then you might get slapped with the title "DevOps Engineer". However:

A job that specialized would be contrary to the idea of DevOps in the first place.
"Jenkins specialist" is a more apt title. Just an example, there are better names for it.

The title is avoided by most companies these days, often replaced by "Automation Engineer". Which, while still vague, it is less so then DevOps Engineer.

Going back to the specifics of the comment, many times the people that installed Jenkins (and the like) in a company were the Systems Engineers. And they probably also explained how it works to other people in the company, maybe even used the term DevOps to explain the philosophy behind it. Which is why the two are often confused. "The guy that takes care of Jenkins? Oh, you mean the DevOps guy? Yeah, DevOps Engineer". No, he's a Systems Engineer practicing DevOps. If nothing else, he should know a lot more than just Jenkins.


Why are books so prevalent in the DevOps community? I have seen quite a few of the blogs I follow recommending more and more books over time.

I enjoy reading fiction and have no aversion to books but where a blogpost can be updated/rewritten when tech moves on these books which are normally ~£20-30 cannot.

Is there a particular quality to DevOps related titles which is missing from the online world or is everyone but me nuts?

In most cases, the recommended books are not about technology. While technology changes, the fundamental principles behind organizations like system thinking, leadership, common sense, etc... do not change as often.

Books such as The Goal, and even The DevOps Handbook do not mention much technology on their pages but rather ways to manage work being performed by people.

Many problems are technology related, topics such as Microservices, Architecting large-scale systems, Infrastructure as Code, etc... these don't talk about a specific tool and/or technology but rather about an architectural topic. A field of knowledge that people who build large systems need to know in order to build the system correctly. This knowledge is rare, and its great the books are written about these subjects - just disregard the tools mentioned, or translate into their new-er reincarnation.

One of the better books about creating quality software (imho) is Agile Software Development, Principles, Patterns, and Practices. And while the language used in this book (Java) has moved along quite a bit, the examples provided in the book are timeless and can be easily translated into any other language of choice.

Some of the problems the DevOps movement tries to solve are involved with common ways work is managed in organizations that just doesn't make any sense. As Eliyahu Goldratt often said (author of The Goal) "Common sense is not very common".

These books teach principles of thinking correctly about problems and human relationships in a system setting so that the whole system is improved. The lessons are old, and unfortunately, only rarely there are people who work in the field who actually learned them.

Naturally, there are also authors who wrote books about such and such fizz-bang tech tool that is new and relevant to the field, like AWS or Docker or Jenkins or whatever and just want to push their book sales ... but I try and exclude these kinds of blog posts from my answer.


Suspend autoscaling processes through Cloudformation template I have the following autoscaling as a Cloudformation template :

  MyAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      DesiredCapacity: 1
      MinSize: 1
      MaxSize: 1
      LaunchConfigurationName: !Ref MyLaunchConfiguration
      AvailabilityZones: !GetAZs

I am struggling with defining suspended processes inside Cloudformation template.

Trying an intuitive approach ( adding it as property) and it does not work at all :

  MyAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      DesiredCapacity: 1
      MinSize: 1
      MaxSize: 1
      LaunchConfigurationName: !Ref MyLaunchConfiguration
      AvailabilityZones: !GetAZs
      SuspendedProcesses:
        - AZRebalance
        - Terminate

Got it from official documentation of UpdatePolicy . Indeed, we can suspend processes by using the UpdatePolicy attribute for the AWS::AutoScaling::AutoScalingGroup resource :

So instead of SuspendedProcesses, the right term is SuspendProcesses.
Also instead of putting SuspendProcesses under Properties, it should be under UpdatePolicy > AutoScalingRollingUpdate .

_

  MyAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      DesiredCapacity: 1
      MinSize: 1
      MaxSize: 1
      LaunchConfigurationName: !Ref MyLaunchConfiguration
      AvailabilityZones: !GetAZs
    UpdatePolicy:
      AutoScalingRollingUpdate:
        SuspendProcesses:
        - AZRebalance
        - Terminate


What does mapping in yml file means? I am not sure what mapping in YAMl file means, can someone advise what does mapping in YAML
 file means?

YAML is a very simple, text/human-readable annotation format that can be used to store data.
YAML keeps data stored as a map containing keys and values associated to those keys. This map is in no particular order, so you can reorder it at will. Each pair is in the format KEY: VALUE
When the value is a text string enclose the value with quotes. Some examples below:

KEY1: 'HELLO'
HOST_NUMBER: 1


How do I get the UUID of a build Mac from Azure DevOps? I'm trying to set up an Azure DevOps pipeline to build a macOS app. I think I have all the necessary steps in place, and it runs as far as I expect it to for now.

However, I now have reached the step where, in order to sign the product, the build machine device must be included in our provisioning profiles, which means we have to add its UUID to the list of devices in Apple Developer.

How do I get this pipeline to use this profile? I imagine a solution might involve one of these:

There's a way to find the UUIDs of every Mac which might run the pipeline
There's a way to restrict that pipeline to always use the same Mac with the same UUID
There's a way to specify/spoof the UUID of the build machine
There's an API or something to register the build machine with Apple Developer at build time
Azure DevOps has a special way to circumvent this part of the build process while still allowing signing
Apple has some special way I don't know about to allow an arbitrary build machine to sign without knowing its UUID beforehand

I've looked around for all these and can't find definitive info for any of them. Does anyone know the best practice for signing an Xcode project pipeline in Azure DevOps using a specific set of provisioning profiles?

Our certs and profiles are stored in Azure's "Secure files" section:

And referenced in the pipeline's build steps:

These steps in the Azure DevOps docs may be helpful: Sign Your Mobile App.

Basically you should be able to generate your p12 and provisioning profile on a mac you control, and then using the "Install an Apple certificate/provisioning profile" tasks you can install the certs onto the mac agent for building.  No UUID should be needed for the agent.  Here's a guide that may be useful (it's targeting iOS, not macOS but the concept is the same).

The way I set this up on a project was to go through Microsoft's AppCenter which had tooling to automatically handle the provisioning profile madness: Deploy Documentation.


Is it possible to give only view permission for Jenkins monitoring-plugin We have the monitoring-plugin installed:
https://github.com/jenkinsci/monitoring-plugin

is there a way to give only view permissions once it is installed?

No :). please see Jenkins issue 14750:
https://issues.jenkins-ci.org/browse/JENKINS-14750


components in kubernetes master node? I'm still a newbie learning the kubernetes and trying to clarify the details.
I've read this document about the components and would like to know  the structure of kubernetes master node:

is it allowed to deploy the kubernetes control plane components on different master nodes in same cluster?
(e.g. deploy "kube-schedulter" on master node A and "kube-controller-manager daemon" on master node B)
Is there options to make HA/FT for the master node?
is it suggested to deploy all the kubernetes control plane components together on a master node no matter what kind of use cases?
does the deployment of master node(s) support both static IP address and DHCP IP address assignment?

I do not see any obstacles. All communication is done through the apiserver over https.
Multiple replicas of the controller and scheduler are allowed, leader election is built in. All you need is load balancing in front of the apiservers.
I find this to work best, since usually the failure domain is a node, rather than a component.
Yes.


Jenkins Pipeline job - remember previous values I have a Jenkins Pipeline which uses Terraform and create a compute environment in GCP. There are like 5 questions in the build i.e.

1) GCP Project Name
2) Number of RHEL Servers
3) RHEL Server names
4) Number of CentOS Servers
5) CentOS Server names

This is a standard build job and users cannot build servers if not using this build.

The problem is that when a user comes back to request more servers, they have to fill up all the details again. They have to remember all the values they passed previously.

So the questions is: How do I make this user friendly without creating a separate build job for each GCP project and hard coding the values.

Depending on how long you keep builds for, the Rebuild option in the left hand sidebar will allow you rebuild a previous run of the job with the parameters intact.

Simply find a previous build of the job that that particular user kicked off last time, click into it, then choose Rebuild from the sidebar.

Again this will only work if Jenkins keeps enough builds. We save two weeks for a particular job and I have a coworker who saves a link directly to x build of the job, then once he rebuilds updates his link to the new one, etc. This way he can just open the link he has saved, click Rebuild, and he's good to go as long as he's inside that two week window.

I believe you can even choose to Keep this build forever for a couple of your "template" builds, making them more easily accessible.


Compare Strings to get Unique values in Jinja2 I have a variable with list of node IP's and its respective Availability Zone.

seed = {
  10.18.13.12 = us-east-1a
  10.18.37.93 = us-east-1b
  10.18.68.147 = us-east-1a
  10.18.21.55 = us-east-1b
}

I will be using this variable in Jinja template but want to filter out the variable based on unique Avalability Zones and get the IP. I want to compare the Avalability Zones to get unique values and have IP from each unique Avalability Zone. For instance I want the output in form of

"10.18.13.12, 10.18.37.93"

or

"10.18.68.147, 10.18.21.55"

Is this possible to perform such operations of comparing strings and get its respective value in Jinja template? Anything would help. Thanks in advance.

Yes, this can be done. The following should do the trick:

"{%- for ip, az in seed.iteritems() %}
  {%- if 'us-east-1a' in az %}
    {%- if firstloop is not defined %}
    {%- set firstloop = 1 %}
{{- ip }}
    {%- else %}
{{- " " + ip}}
    {%- endif %}
  {%- endif %}
{%- endfor%}"

To test it with https://cryptic-cliffs-32040.herokuapp.com/, you can use the following JSON:

{
    "seed": {
      "10.18.13.12": "us-east-1a",
      "10.18.37.93": "us-east-1b",
      "10.18.68.147": "us-east-1a",
      "10.18.21.55": "us-east-1b"
    }
}

You might give the documentation a read if you are going to be doing much jinja templating. I have found it indispensible.


`UnauthorizedResourceAccessException` when calling `aws workdocs create-user` When calling aws workdocs create-user, I'm getting UnauthorizedResourceAccessException or UnauthorizedOperationException.  I believe I have all of the permissions required.  What could be causing this?

While the WorkDocs API describes create-user as creating a user in the Directory Service (and does), it also activates the user as a WorkDocs user.  As a result, it can only be called when the directory is associated with WorkDocs (i.e. when the directory is "enabled" in WorkDocs).  These errors can occur when WorkDocs is not enabled.
These errors can arise, for example, if you're trying to create users for WorkSpaces using the WorkDocs API since a comparable API isn't available for WorkSpaces.  To enable WorkDocs, do one of the following:

Use EnableWorkDocs when registering the Directory with WorkSpaces
Manually create a site on the WorkDocs page

WARNING:  Even if you allocate no storage, it appears these users will incur WorkDocs charges unless you also deactivate_user.
WARNING:  EnableWorkDocs is not fully reversible from the API i.e. you cannot delete the Directory Service from the API if a WorkDocs site exists and the site cannot be removed from the API (likely for data protection reasons)


Commit messages from Submodules to Jenkins Change Log I have a project with 3 submodules that are updated whenever that project is built and the project is built mainly for an installer (which is not in a submodule). Not a lot of changes are made in the installer itself but the commit messages from there are coming through fine.

I'd like to also see the commit messages (changes made) from the submodules to the Jenkins log. Is this possible? If not possible there, would it be possible in the email-blast instead? In the mailer plugin, I have it configured as:

${CHANGES, showPaths=true, format="Commit: %r by %a \n %p \n\t Comment: %m \n", pathFormat="\n\t %p"}

Also, I kinda inherited Jenkins from an employee that no longer works where I work so I'm very new to this.

After consulting this cheatsheet, I think that you are looking for the subject. If I apply:

git log --pretty="%s"

to one of my git repositories, the following is returned:

learning pointers in golang
receiver struct in golang
embedded structs in golang
multiple spanish words

In conclusion, I think you should include %s to get the git messages or basically the commit subjects.


Kubernetes - How to show all service accounts How do I show all service accounts In Kubernetes?

I have tried kubectl get --all-namespaces all. It does not show service accounts.

How can I use kubectl to list all service accounts?

The command you listed will show you your resources.

Instead try: kubectl get serviceAccounts


What strategies can be employed to secure sensitive data in log files? Working in highly regulated environments data is classified in different ways depending on the sensitivity. In some cases, this is legally enforced and must be treated differently.

Examples of a data classification policy are:

Highly Restricted Data, such as Passwords, Private Keys, SAML Tokens, and Credit Card Numbers.
Restricted Data, such as Usernames and Customer IDs.
Unrestricted Data, pretty much anything else.

This classification comes with certain obligations:

Any data that is Highly Restricted must never be made available in log file under any circumstances.
Restricted data could be made available in log files under specific conditions. For example, if an incident were to occur with a service then the on-call engineer would be able to enact a Break-Glass procedure to access this data to diagnose the problem. The Break-Glass procedure would, in turn, trigger a review, an audit, and possibly temporary revocation of privilege from that engineer.

What strategies can be employed to achieve this, particularly given that there are a broad range of logging, monitoring and instrumentation tools available on the market that don't provide a direct answer to this problem?

For example, both Splunk and AppDynamics have the ability to impose different access controls upon conditions of the telemetry being exposed; this means you can create a filter that masks out <customerId>NNNNNNNNNNNN</customerId>.  However, neither of these tools give you the capability automatically identify credit card numbers and mask them automatically.

Note: I believe this is related to DevOps because in a traditional tiered support model a relatively small group of people could have access to the data and filter it manually, by devolving responsibility for operating platforms to development teams this data is potentially exposed to a far wider audience.

I think the solution comes down to a broad spectrum of approaches that ensures data protection:

Data Classification: The most efficient technical strategy is to categorise the data at the point of creation rigorously. At its core, the developers are responsible for ensuring that all logged information is assigned a category.  Categorization can, for example, be achieved through Splunk Metadata which, in turn, can be used to direct log entries to different buckets based upon their data categorization.
Event Partitioning: There is often a desire to log sensitive information along side non-sensitive information. For example, if a new user was to sign up you may log:

Customer ID (Restricted)
Customer Type (Restricted)
Aquisition Source (Unrestricted)
Correlation ID (Unrestricted)

It is possible to split this one "Event" into two parts, one containing the Restricted information and one containing th Unrestricted information.  This aligns with the first point by allowing filtering rules to direct to different buckets.
Data Masking: In specific circumstances it may not be possible to categorise the data at source, in my experience logging solutions do allow for Masking Rules to X out sensitive data.  In the linked example a sed command is used to apply a regular expression to all data from a specific source.  Once the restricted data has been masked out then the event can be considered to be Unrestricted.  Care must be taken with the rule to ensure that information critical to an event such as a correlation ID doesn't match the Regular Expression used to match sensitive data.
Event Filtering: As a last resort it may be necessary to filter all events of a particular type or source into a separate bucket if they contain sensitive data that cannot be categorised or masked.  In this case the information in a restricted bucket could only be accessed through a Break-Glass mechanism to bypass access controls under the provisions of an incident.

With each of these solutions, testing is key to ensuring that nothing slips through the cracks.  Logging and Event Management must be considered to be a first class Non-functional Requirement with the same level of development rigour applied to a solution - including peer review and tests to ensure that data is properly categorised and partitioned in the tool of choice.


How do you calculate the compound Service Level Agreement (SLA) for cloud services? Cloud services hosted by Amazon Web Services, Azure, Google and most others publish the Service Level Agreement, or SLA, for the individual services they provide.  Architects, Platform Engineers and Developers are then responsible for putting these together to create an architecture that provides the hosting for an application.

Taken in isolation, these services usually provide something in the range of three to four nine's of availability:

Azure Traffic Manager: 99.99% or 'four nines'.
SQL Azure: 99.99% or 'four nines'.
Azure App Service: 99.95% or 'three nine five'.

However when combined together in architectures there is the possibility that any one component could suffer an outage resulting in an overall availability that is not equal to the the component services.

Serial Compound Availability

In this example there are three possible failure modes:

SQL Azure is down
App Service is down
Both are down

Therefore the overall availability of this "system" must lower than 99.95%.  My rationale for thinking this is if the SLA for both services was:

  The service will be available 23 hours out of 24

Then:

The App Service could be out between 0100 and 0200
The Database out between 0500 and 0600

Both component parts are within their SLA but the total system was unavailable for 2 hours out of 24.

Serial and Parallel Availability

In this architecture there are a large number of failure modes however principally:

SQL Server in RegionA is down
SQL Server in RegionB is down
App Service in RegionA is down
App Service in RegionB is down
Traffic Manager is down
Combinations of Above

Because Traffic Manager is a circuit breaker it is capable of detecting an outage in either region and routing traffic to the working region, however there is still a single point of failure in the form of Traffic Manager so the total availability of the "system" cannot be higher than 99.99%.

How can the compound availability of the two systems above be calculated and documented for the business, potentially requiring rearchitecting if the business desires a higher service level than the architecture is capable of providing?

If you want to annotate the diagrams, I have built them in Lucid Chart and created a multi-use link, bear in mind that anyone can edit this so you might want to create a copy of the pages to annotate.

After reading Tensibai's excellent answer, I realised I used to be able to calculate this for network analysis purposes.  I dug out my copy of High Availability Network Fundamentals by Chris Oggerino and had a crack at working this out from, not quite first principals.
Taking my serial example directly out of Tensibai's answer is simply a case of multiplying the probability of each component being available by the other:

So

99.95% * 99.95% = 99.9%

Calculating it in parallel is a little more complicated as we do need to consider what the percentage unavailability will be:

The calculation is done as follows:

Multiply the unavailability of the two regions together.

0.1% * 0.1% = 0.0001%

Convert that back to availability

100% - 0.0001% = 99.9999%

Multiply the Traffic Manager availability by the availability of the two regions.

99.99% * 99.9999% = 99.9899%

The result is the whole system availability.

99.9899% is close to 99.99%

I ended up using Excel to perform the calculations, here is the values:

... and the formulas ...


What topics should at least be included in a Java for DevOps course? I am preparing a course about "Java for DevOps". From a DevOps perspective, what topics should be covered during the meeting?

I was thinking about the following topics:

What is DevOps?
Why should I learn Java or a programming language that the team is using from a DevOps perspective?
Java itself

keywords
keyword categories
OOP
FP
Hello World

The aim of this question is to get feedback. On one hand I think that these topics will cover "Java for DevOps", but on the other hand I would like to know whether other people agree with this or what topics they would cover if they will organize the course?

The target audience consists of Senior/Medior Quality Assurance, Senior/Medior Operations and some Junior Developers.

Rationale

According to Wikipedia, DevOps is the intersection of Development, Operations and Quality Assurance.

Development: creates features, i.e. add business value
Operations: ensures that the software runs stable
Quality Assurance: tests the software by clicking in the UI for example

DevOps: intersection, i.e. automated, stable, tested software sooner to the customer

As development creates new feature, it would be beneficial for the company and also the developers that Testers and Operations automate more and start to fix bugs, then Development could create more features.

If everything goes well, perhaps some testers and operations could create features in a couple of months/years as well.

Even with the rationale you edited in, I don't really see a point.

DevOps is an intersection of Dev, Ops (and more, like Security and maybe Testing) in a cultural sense. But on a technological sense, you will always have people in a mixed team who are good at one of those but not the others, and that is fine.

For example: you might deploy something like Kubernetes or OpenShift; and then you will for sure have ops-sided people who are expert at running the low-level stuff, tuning your NAS, adapting your orchestration to a cloud provider of your choice, or to on-premise bare-metal servers and so on. You will not, usually, have all of your developers do the same; they may use those things provided by the others. The ops guys might never see a piece of application code at all; and you would still call it "DevOps".

What your devs will be able to do is manage the full lifecycle of their applications, from source code to production, using the tools provided by the ops guys. When one says that devs do jobs that ops used to do in the past, one means that the automation makes it so that the environments (dev, test, int, prod...) are mostly the same; that deploying to them is as automated as humanly possible (without requiring manual ops, and most importantly without requiring a handover of binaries etc. between devs and ops); that there is a complete, unbroken technological chain from source to prod; including things like virtual networks automatically springing into existence and so on. One does not mean that devs troubleshoot hardware or networking on a low layer.

The same goes for your ops guys. They will most certainly not try to change the application code proper after a few seminars on it, unless you are talking about really small companies with 3-person teams who need to do so out of sheer necessity.

Note that I am not saying that it is not possible to be one person who does both development and operations work. Is is an absolute gold mine to have such people available; and some companies foster that by, for example, rotating devs into low-level ops groups occasionally. But those guys are not your target audience. By the topics you listed in your question, you are targeting ops guys who have never in their life coded anything ("hello world").

If you actually want your ops guys to grow into coding instead of just clicking around in GUIs, modifying or creating config files and such, then here are some suggestions:

If they are writing the occasional bash script already, point out advances constructs to them (loops, functions, error handling, etc.).
Try to get them interested in some dynamic language like perl or ruby for a low-hanging fruit, one step above shell scripts.
Introduce them to go, which is really nice for standalone command line applications and especially automation tools. It should satisfy the need to learn something new which should go really well on anyone's resume, and will satisfy the naysayers regarding non-natively compiled languages.

... and so on. But don't try to sell them "Java" as a route into "DevOps".