Terraform - how to modify a existing S3 bucket that was created from module? I already have my S3 bucket called foo-name created and existing with terraform. I want to modify the existing s3 bucket and a policy. But when I do this it seems to want to add a new s3 bucket instead. I imagine s3_website in resource "aws_s3_bucket" "s3_bucket" is incorrect. But I don't know what I would use in place of that because of the way the S3 bucket was corrected.

How do I get it to modify my existing S3 bucket that was created in module s3_website {}?

provider "aws" {
  alias = "east1"
  region = "us-east-1"
}

module "acm_cert" {
  source = "foo-git"

  cluster = var.cluster

  domain      = var.website_domain
  dns_zone    = var.zone_name

  tags       = var.tags
  extra_tags = var.extra_tags

   providers = {
    aws = aws.east1
  }
}

module s3_website {
  source = "foo-git"

  cluster        = var.cluster
  env            = var.tags.Environment
  app            = var.tags.Application
  website_domain = var.website_domain
  s3_prefix      = var.s3_prefix

  certificate_arn = module.acm_cert.certificate_arn
  error_document = "index.html"
  index_document = "index.html"

  custom_error_response = [
    {
        "error_code"         = 404
        "response_code"      = 200
        "response_page_path" = "/index.html"
    },
    {
        "error_code"         = 403
        "response_code"      = 200
        "response_page_path" = "/index.html"
    },
  ]

  tags = var.tags
}

data "aws_route53_zone" "base" {
  name = var.zone_name
}

resource "aws_route53_record" "this" {
  zone_id = data.aws_route53_zone.base.zone_id
  name    = var.website_domain
  type    = "A"

  alias {
    name                   = module.s3_website.domain_name
    zone_id                = module.s3_website.zone_id
    evaluate_target_health = false
  }
}

resource "aws_s3_bucket" "s3_website" {
  bucket = "foo-name"
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GETObject",
      "Resource": "arn:aws:s3:::foo-name/*"
    }
  ]
}
POLICY
}

You'll have to modify the underlying module to make changes here.

Be sure you're aware of the impact this will have on other modules using your s3_website module. You might want to start using module source versioning.


Circle ci workflow job stuck in "created" when adding a required step to the workflow workflows:
  version: 2
  build_deploy_test:
    jobs:
      - build:
          filters:
            branches:
              only:
                - develop
      - christest:
          requires:
            - install
          filters:
            branches:
              only: devops-docker-intial

With the following workflow a commit to the devops-docker-intial branch just results in a workflow that is 'created' - it doesn't proceed further... Why?

Due to the fact that build had a branch filter... Even though no error message on additional info is shown...

Hence I needed to add the branch like so:

workflows:
  version: 2
  build_deploy_test:
    jobs:
      - build:
          filters:
            branches:
              only:
                - develop
                - devops-docker-intial
      - christest:
          requires:
            - install
          filters:
            branches:
              only: devops-docker-intial


Secure deployments of on-prem artifacts to cloud Right now, we have locally hosted (across multiple Datacenters) artifactory and Bitbucket. We're at a halfway point between Jenkins and Bamboo (with Jenkins beginning to make more traction against Bamboo).

However, we're currently going through the task of building out a hybrid cloud platform, which is essentially going to be K8s on top of a cloud provider such as AWS or Azure. However, I've no idea how to approach securely deploying code or artifacts over the internet. Nor can I seem to find a 'best practice' cheat sheet.

Any pointers?

Both Azure and AWS will have any container images built off their network pushed to their repositories using https which will encrypt the transfer (e.g. AWS ECR encryption).  Both of these providers offer ways to encrypt the transfer of data into their non-container storage solutions (often via https) and then offer ways to encrypt the data at rest.  Once it is on the cloud provider deploying to k8s can all be done securely within their servers.


pre-DevOps deployment metrics challenge TL;DR, how do you prove devops, specifically deployment automation, improves change failure rates?

We're all trying to capture metrics on 'deployment failures' using current (mostly manual) means. Unfortunately, a 'failure' rarely happens, right? Because when something goes wrong, the team comes together (typically with heroics) to fix the issue (typically permissions, missed configs, you know the drill). So... when we ask how the deployment went, the answer is "it worked."

But, intuitively we all know that's not good. The 2017 state of devops report says there's about a 31-45% "change failure rate." While that intuitively sounds about right, are they tracked as incidents? Nah. Because they get fixed pretty quickly, usually during validation. It's much more rare to actually roll back a deployment.

So, it takes discipline to report failure rates accurately. We're disincentivized to report like that because we want things to work and we do what it takes to make it happen.

So, how do you prove devops, specifically deployment automation, improves change failure rates?

(PS tried to tag this with "#devops-capability-model")

A technique we've used in the past in similar situations is to get "management commitment" that imposes these rules to every team member:

Access to perform updates to the target deployment areas (i.e. production) is limited to selected automated systems, which have appropriate audit trails (= logging) of any kind of updates to the areas they manage.
Manual updates to the target deployment areas, for whatever reason, are no longer allowed by the typical team members (user ids) that used to be able (authorized) to perform these updates. Instead NEW (additional) user IDs will be created which will have all required permissions to (still) perform such manual updates, whenever needed. But to actually be able to use those new user IDs (= perform a logon with them), the team member who wants to perform a logon with such new user ID will have to perform "some" extra step to get access to the password for such new user Id. Ideally this extra step is automated also (use your own imagination how it should look like), but if anything else fails: just contact (= eMail, call, etc) the gate-keeper of the required password, including  "which issue they have to get fixed" (similar to your question).
Getting such gate-keeper in place is not an easy job. And the most resistance will come from ... the team members (for all sorts of reasons). Therefor a variation of those new user IDs (as in the previous step) is that each team member gets an extra user ID (with the password they decide themselves), but with an extra string attached to it: they are only allowed to perform a logon with that (extra) user ID if they actualy have a good reason to do so. And each time they perform such logon, they are required to file some type of report about "which issue they fixed" (similar to your question).

With these procedures in place, all that's left to do is to periodically review each of those reports / reasons why it was required to use such special user ID, and ask the question "Is there anything that can be done to further automate this, to further reduce the need for such special login?".

Update:

Quote from your extra comment below this answer:

  I think adding artificial barriers to fixing a deployment issue is counter-productive.

True it adds an extra barrier, but I'm not conviced it is "artificial". Because this is, to my knowledge, the only way to become aware of things those team members otherwise won't ever tell you, for reasons such as:

job security.
bad things/practises they prefer to keep hidden.
power they do not want to loose.


Gitlab CI/CD: deploy docker image to existing kubernetes cluster I have node.js application that I need to deploy to exising kubernetes cluster.
The cluster is setup using kops on AWS.
I have created .gitlab-ci.yml file for building docker images.
So, whenever a change is pushed to either master or develop branch. It will build the docker image.
Now, I have to deploy it  to exisitng kubernetes cluster. I have already followed steps mentioned here to add existing cluster.
For now, I am using shared runner.
How can I integrate kubernetes deployment after image is built with gitlab ci/cd to deploy on aws (cluster is created with kops)?
For registry I am using gitlab's container registry not docker hub.
I have tried below gitlab-ci.yaml file,
gitlab-ci.yaml
stages:
  - docker-build
  - deploy

docker-build-master:
  image: docker:latest
  stage: docker-build
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE:prod" .
    - docker push "$CI_REGISTRY_IMAGE:prod"
  only:
    - master

deploy-prod:
  stage: deploy
  image: roffe/kubectl
  script:
    - kubectl apply -f scheduler-deployment.yaml
  only:
    - master

docker-build-dev:
  image: docker:latest
  stage: docker-build
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE:dev" .
    - docker push "$CI_REGISTRY_IMAGE:dev"
  only:
    - develop

I am getting below error.
Executing "step_script" stage of the job script
00:01
Using docker image sha256:c8d24d490701efec4c8d544978b3e4ecc4855475a221b002a8f9e5e473398805 for roffe/kubectl with digest roffe/kubectl@sha256:ba13f8ffc55c83a7ca98a6e1337689fad8a5df418cb160fa1a741c80f42979bf ...
$ kubectl apply -f scheduler-deployment.yaml
error: unable to recognize "scheduler-deployment.yaml": Get http://localhost:8080/api?timeout=32s: dial tcp [::1]:8080: connect: connection refused
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1

File scheduler-deployment.yaml is a valid yaml file and it works when I apply manually.
Below is the deployment file.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sheduler-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: scheduler
  template:
    metadata:
      labels:
        app: scheduler
    spec:
      containers:
        - name: scheduler
          image: registry.gitlab.com/user/schedulingapis:dev
          env:
            - name: REDIS_HOST
              value: "redis://redis-master "
            - name: RABBITMQ_HOST
              value: "amqp://rabbitmq"
      imagePullSecrets:
        - name: registry-credentials

If someone has any better idea to automate deployment to a k8s cluster with gitlab.
Pease do suggest.
Any link or reference will also work.

Your error is most likely from referencing the wrong directory in your script block of your job.
When GitLab executes your job in a container, it mounts the project directory to the /builds directory in the container. You can reference this either by specifying /builds/scheduler-deployment.yaml or by using the predefined environment variable $CI_PROJECT_DIR with $CI_PROJECT_DIR/scheduler-deployment.yaml.


Chef - assign a value to variable in execution or converge phase I am creating a recipe that has two resources. The initial VPN IP address is a empty valued string. Upon execution of the first resource, a IP address within the range 10.12.xx.xx is set. Then a second resource is executed with a guard block that checks for the VPN IP.

#
# Cookbook:: test_cookbook
# Recipe:: check-vpn-ip
#
# Copyright:: 2019, The Authors, All Rights Reserved.

#Getting the IP address using the ruby's Socket class.

ip_list = Socket.ip_address_list
vpn_ip_list = ip_list.select{ |ip| ip.ip_address.match(/^10.12/) }
!vpn_ip_list.empty? ? ip_addr = vpn_ip_list.first.ip_address : ip_addr = ""

execute 'manually_start_open_vpn' do
command "sudo openvpn #{node['openvpn-conf-path']}/#{host}.conf &"
action :nothing
only_if {ip_addr.length.eql?(0)}
end

new_ip_list = Socket.ip_address_list
new_vpn_ip_list = new_ip_list.select{ |ip|
                                   ip.ip_address.match(/^10.12/) }
!new_vpn_ip_list.empty? ? newvpn_ip_addr =
                new_vpn_ip_list.first.ip_address : newvpn_ip_addr = ""

ruby_block 'chat-bot' do
   block do
        machine_data = {text: "OpenVPN IP not assigned to #{host}
            software_version: 18.4.4 \n This is a test message please
              ignore @all"}.to_json
        header = {'Content-Type': 'text/json'}
        http = Net::HTTP.new(google_chat_uri.host,
                   google_chat_uri.port)
        http.use_ssl = true
        request = Net::HTTP::Post.new(google_chat_uri.request_uri, header)
        request.body = machine_data
        response = http.request(request)
    end

   end
only_if {newvpn_ip_addr.length.eql?(0)}
action :nothing
end

Since there is no vpn ip address of the range 10.12.xx.xx during compile phase, the value of ip_addr.length.eql?(0) is true and the guard block condition in first resource is satisfied and it's executed.

As the vpn ip has already been assigned, the newvpn_ip_addr.length.eql?(0) should be false and the 2nd ruby block should not be executed. But this returns a false value as the code newvpn_ip_addr is executed during compile time itself.

Is there any way to make sure that the value for newvpn_ip_addr in the below code is assigned and checked during the execution phase post the execution of first resource?

    new_ip_list = Socket.ip_address_list
    new_vpn_ip_list = new_ip_list.select{ |ip|
                                   ip.ip_address.match(/^10.12/) }
    !new_vpn_ip_list.empty? ? newvpn_ip_addr =
                new_vpn_ip_list.first.ip_address : newvpn_ip_addr = ""

So that the guard block in 2nd resource is not executed if the expected IP address is not assigned.
only_if {newvpn_ip_addr.length.eql?(0)}

You have a problem of compile vs converge times as you guessed.

You code between the execute resource and the ruby_block resource is executed before the execute resource is converged.

The easiest way to circumvent that is to put that code within a ruby_block, and to avoid scoping problem use node.run_state hash to store the value to be checked.

A rewrite of your recipe which should work would be:

#
# Cookbook:: test_cookbook
# Recipe:: check-vpn-ip
#
# Copyright:: 2019, The Authors, All Rights Reserved.

#Getting the IP address using the ruby's Socket class.

ip_list = Socket.ip_address_list
vpn_ip_list = ip_list.select{ |ip| ip.ip_address.match(/^10.12/) }
!vpn_ip_list.empty? ? ip_addr = vpn_ip_list.first.ip_address : ip_addr = ""

execute 'manually_start_open_vpn' do
  command "sudo openvpn #{node['openvpn-conf-path']}/#{host}.conf &"
  action :nothing
  only_if {ip_addr.length.eql?(0)}
end

ruby_block 'check_vpn_ip_list' do
  block do
    new_ip_list = Socket.ip_address_list
    new_vpn_ip_list = new_ip_list.select{ |ip| ip.ip_address.match(/^10.12/) }
    node.run_state['newvpn_ip_addr'] = !new_vpn_ip_list.empty? ?  new_vpn_ip_list.first.ip_address : ""
  end
end

ruby_block 'chat-bot' do
  block do
    machine_data = {text: "OpenVPN IP not assigned to #{host}
        software_version: 18.4.4 \n This is a test message please
        ignore @all"}.to_json
    header = {'Content-Type': 'text/json'}
    http = Net::HTTP.new(google_chat_uri.host,                       google_chat_uri.port)
    http.use_ssl = true
    request = Net::HTTP::Post.new(google_chat_uri.request_uri, header)
    request.body = machine_data
    response = http.request(request)
  end
  only_if {node.run_state['newvpn_ip_addr'].length.eql?(0)}
  action :nothing
end

There's no real use of response = in the last line as you don't use it later, but I let it as is in case you wish to log in the ruby block the response.


GitLab CI/CD: How do I generate a change log between pipeline runs? I'm currently using Gitlab CI/CD combined with Fastlane on our build machine to perform nightly builds of an iOS app, and push those builds to TestFlight on completion. So far, so good. However, I've been asked to generate a change log of the new commits that have gone into each night's build.

Is it possible to get not just the CI_COMMIT_SHA of the current pipeline run, but also the SHA of the previous commit that the pipeline last ran for? That way, I could generate a change log of what's new in the current pipeline run? After searching the docs on Gitlab.com, I'm coming up empty so far.

As part of the pipeline, we also generate a build manifest that includes the current commit SHAs used for the build, and save that as an artifact. So, if it was possible to download the most recent artifacts, I could also maybe just look at that manifest, but again, I'm not seeing a way to do that either.

Note: the answer is rather speculative, based on docs only, I didn't actually use GitLab pipelines yet.
You may be able to use the Pipelines API. More specifically you could get a pipeline's variables:

Get variables of a pipeline
GET /projects/:id/pipelines/:pipeline_id/variables

Attribute Type            Required    Description
id            integer/string  yes         The ID or URL-encoded path of the project owned by the authenticated user
pipeline_id   integer         yes         The ID of a pipeline

Example of response
[
  {
    "key": "RUN_NIGHTLY_BUILD",
    "variable_type": "env_var",
    "value": "true"
  },
  {
    "key": "foo",
    "value": "bar"
  }
]

I'm suspecting that in the response you'd find the variables shown in a pipeline creation response example, which include these which could be of interest to you:

  "id": 61,
  "sha": "384c444e840a515b23f21915ee5766b87068a70d",

  "before_sha": "0000000000000000000000000000000000000000",

I'm uncertain if the before_sha for the current pipeline is what you're looking for, if not you'd just get the sha for the previous pipeline ID, which you could derive from the current pipeline ID (via the CI_PIPELINE_ID environment variable, see Predefined environment variables). I'm assuming that the pipeline IDs are consecutive.


Are there serious problems with an EC2 auto-scaling AMI that automatically downloads newest server from git? I'm converting some servers from manual scaling (with a load balancer) to auto-scaling. I'm currently using an Ansible  script to manage upgrades, and I wouldn't want an automatically created instance to not be using the newest version of the software. I created an AMI that will download the newest commit from a chosen branch on first boot.
I've since realized my approach is somewhat like "cowboy coding", since AWS provides a mechanism to replace auto-scaling instances when there should be an update. (If using that, my existing update script would become obsolete, and I guess the update would entail creating a new AMI containing the new server version.)
Are there any serious problems with using the "cowboy" approach? I realize auto-created servers might end up with newer code than the other servers (if someone pushes but does not deploy their code changes). I wonder whether auto-scaling will cause Ansible scripts to fail if servers are terminated while being processed by Ansible. What have I missed?

What you're going to do is only calling for inconsistencies between your nodes.
I wouldn't do any deployment during the instance boot-up, instead install AWS CodeDeploy as part of your boot-up (or bake it into the AMI) and once the instance boots up it contacts the CodeDeploy service and obtains the newest code version.
Likewise if you need to roll-out a newer code version push it to CodeDeploy and it will take care of updating all the servers in your Auto-Scaling Group in a consistent, predictable way.
With ASG you really really want to prevent inconsistencies between the nodes and this is one of the ways to do it.
Hope that helps :)


Checking Jenkins build status from only one pipeline and avoid from other pipeline pointing to same gitlab repo in PR build Hi I am having two jenkins pipeline pointing to same git repo one pipeline  is named as mainpipeline and other is sit pipeline. Before sit pipeline our git PR build is checking the build status from mainpipeline. But after introducing sit pipeline which is pointing to same git repo as mainpipeline, our git build status is start checking from sit pipeline . How should i avoid checking the git build status from the sit pipeline and check only from mainpipeline.
I have suppress the build check status in sit pipeline but sometimes its working and sometimes not below is the screen shot of configuration

As far as I can tell from reading the source code, the build status notifications are basically hard-coded into the GitLab Branch Source plugin and cannot be disabled.  Therefore, without writing your own plugin or something like that, the only way I can think of to disable the build status notifications would be to use a different method of checking out the repository that does not use the GitLab plugin, e.g. using the checkout step with a direct url.


How to version control shared database? We have several C# applications using a shared monolithic database (on-premise SQL server) using Stored Procedures as an exclusive interface.
The C# code repositories include only Stored Procedures that are relevant to them, which means a single database is split into multiple repositories.

We are currently using a shared DEV environment restored from production (after some modification to sensitive data, etc.)

I want to move to our version control being a single source of truth and put the whole database into one repository so I can generate empty DB for each developer (that would allow me to unit test, etc.)

But that would create problems for developers: they would have to keep a branch of their app and branch of a database in sync and coordinate extensively.

How to solve this without moving the apps back into monolith as well?

I personally made the unwise choice of cramming five different apps into a single database about 20 years ago. I then had to deal with the repercussions of that. The bad news is, this is really hard. The good news is, modern tools make it easiser, but not easy.

So, first, absolutely get a single source of truth. You're using SQL Server, so there are a bunch of really good tools that will help you maintain your database in source control. One is provided by Microsoft. It's called SQL Server Data Tools. It only works inside Visual Studio. However, it's a solid tool. If you have an MSDN license to run VS, it's complete free. There are also 3rd party tools available for a cost. They're generally much more sophisticated (Microsoft doesn't give SSDT much love). I work for one of the vendors of such tools, Redgate Software.

Once you're on the path of maintaining the code in source control, the easy part is over. Now you have to figure out how to integrate the teams. Here's what we did. Every team got it's own branch of code (we did this in the old TFS, which sucked, you can use some variant of Git which will be much easier, not easy, just easier). This also means that each team had to have it's own copy of the database (which it sounds like what you're going for anyway). In fact, each developer got their own database and then  we had an integration testing database for each team. So the team worked on it's own, independent of the other teams.

Next, we created a master integration database. This is where the teams came together. We had a main branch for the code and teams would have to merge their code semi-regularly into this branch and then we could run automated tests to see if any team had broken the code or structures of other teams.

Then, you have to build a release process that ensures that any one team can't overwrite or supersede the others. This means that teams have to regularly merge from the main branch, changes from the other teams.

It requires quite a bit of discipline and some training to get this all set up. However, it's doable. I'd also suggest implementing a flow control software like Jenkins, Octopus, AWS DevOps or Azure DevOps to assist with automating all of it.

I've worked on tons and tons of documentation around these types of processes. Full disclosure, a lot of the info at that link includes our tooling. However, the processes are what you're looking for and you can find good information there. This is a free ebook on these processes I helped to write that's very product agnostic. Hopefully between these resources you get enough to help out.


How to get the latest Tomcat version? If one checks the Tomcat homepage then one could see some versions, but not only the latest. Off course I could parse the website and find the version, but I am looking for a Tomcat API that shows the latest version to prevent that I have to create a script that is error prone.

Why?

I would like to automate the update of our internal tomcat packages. At the moment someone is checking the latest version and create the debian package manually, but I want to see that that is automated.

What has been tried?

The github mirror does not show the latest version https://github.com/apache/tomcat/releases
The official tomcat html could be parsed, but that does not seem to be the easiest solution
curl https://api.github.com/repos/apache/tomcat/releases/latest returns:

{
  "message": "Not Found",
  "documentation_url": "https://developer.github.com/v3/repos/releases/#get-the-latest-release"
}

Get the latest version of Tomcat:

$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=1' | jq -r .[].name
9.0.20

Get the latest version of Tomcat 9:

$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=100' | jq -r '.[] | .name' | sort -V | grep ^9 | tail -n 1
9.0.20

Get the latest version of Tomcat 8.5:

$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=100' | jq -r '.[] | .name' | sort -V | grep ^8.5 | tail -n 1
8.5.41

Get the latest version of Tomcat 7:

$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=100' | jq -r '.[] | .name' | sort -V | grep ^7 | tail -n 1
7.0.94

These all currently match the latest versions listed at http://tomcat.apache.org/


Where is the apache2 filebeat module configured? I am setting up the Elastic Filebeat beat for the first time.  I have followed the guide here, and have got the Apache2 filebeat module up and running, it's connected to my Elastic and the dashboards have arrived in Kibana.

Currently it's using the default path to read the Apache log files, but I want to point it to a different directory.

There is a setting var.paths documented here for this purpose, but I can't see where this setting is applied in the configuration for Filebeat.

I've looked through the Yaml files in the installation and can see the Apache2 module default config, but it doesn't look like I should modify that.

I'm guessing I should add something to the filebeat.yml, but there doesn't seem to be a placeholder for "modules.apache" or similar.

I'm trialling on MacOSX, will be deploying on Ubuntu, but I don't believe this to be a platform-specific question.

After looking a bit further, I found it in the sample called filebeat.full.yml

So I added this section to the filebeat.yml
Restarted filebeat

After running at 100% CPU for minute, Kibana is now immediately showing the results. That was easier than I expected.

#==========================  Modules configuration ============================
filebeat.modules:

#------------------------------- Apache2 Module ------------------------------
- module: apache2
  # Access logs
  access:
    #enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["MY-CUSTOM-PATH"]


How to get mobile push notifications on Jenkins jobs? I'd like to receive automatic push notification to my mobile about completed Jenkins jobs and their status (success or failure) in any form. Is it possible?

Jenkins has a bunch of plugins:

There's HTML5 desktop notifier plugin which can push notifications to your browser
There's a Slack plugin which can push notifications to a Slack team/channel


Auto create Jenkins jobs for branches and pull request via declarative code in scm I am looking to auto create jobs in jenkins upon pull request , branches, master push etc similar to what we do in Gitlab. My SCM is butbucket here.
I have so far setup docker based agent integration with Jenkins and butbucket, when I create a job and configure it to use repo it all works fine , but I just want to remove altogether a step of job creation in jenkins and want the workflow like this:
In butbucket source code repo to keep all pipeline configuration for and branch and tag to trigger Jenkins pipeline without touching Jenkins for job creation or any config creation. Just want to drive all via the script in code repo for pipeline .
Any recommendations or help for workflow would be appreciated

Use the Bitbucket Branch Source plugin, which is an official plugin developed by the Jenkins developers.  This plugin automatically creates jobs for all branches and pull requests of all repos in a project, and if you use Bitbucket Cloud it will also set up webhooks in Bitbucket to ensure pushes and PR updates automatically trigger builds in Jenkins.


What is the difference between dockerd and com.docker.service in Docker for Windows? What is the difference between dockerd and com.docker.service in Docker for Windows? Or rather I should ask: why do I need (if I need) com.docker.service running?

Intention: I want to use Docker only with command line in Windows environment in Windows container mode, control it with PS scripts etc. etc.

For me the role of com.docker.service is unclear.

I am able to run dockerd process and pull and run containers, when com.docker.service is stopped. When I start it - nothing happens (or maybe I don't notice the change).

dockerd is a persistent process that manages containers. Docker uses different binaries for the daemon and client. ( from Docker site ). Docker Daemon is a server which interacts with the operating system and performs all kind of services. The Docker Daemon listens for REST API request and performs the operation. Its process running on the background.

Docker service is the utility we use when we run any docker commands e.g. docker run (docker container run), docker images, docker ps etc. It allows us to run these commands which a human can easily understand.


Compatibility of http-builder-ng Groovy module with Jenkins 2 Did somebody had a chance to use http-builder-ng Groovy module in Jenkins pipeline?

When I use this module with any kind of request/URL it always returns me null object when the same source code from my Desktop is being run fine.

Here is debug output:

<groovyx.net.http.UriBuilder$Basic@4bc2413c scheme=http port=-1 host=api.open-notify.org path=/astros.json query=[:] fragment=null userInfo=null parent=groovyx.net.http.UriBuilder$ThreadSafe@69c6847a useRawValues=null>

The older version of this module developed by groovy.codehouse works fine, but it doesn't have some methods which I need.

I would recommend against importing if at all possible. You can use built-in functionality to achieve the same results. Also be aware of "In-process Script Approval"

https://stackoverflow.com/a/42662243/1678094

// GET
def get = new URL("https://httpbin.org/get").openConnection();
def getRC = get.getResponseCode();
println(getRC);
if(getRC.equals(200)) {
    println(get.getInputStream().getText());
}

// POST
def post = new URL("https://httpbin.org/post").openConnection();
def message = '{"message":"this is a message"}'
post.setRequestMethod("POST")
post.setDoOutput(true)
post.setRequestProperty("Content-Type", "application/json")
post.getOutputStream().write(message.getBytes("UTF-8"));
def postRC = post.getResponseCode();
println(postRC);
if(postRC.equals(200)) {
    println(post.getInputStream().getText());
}

You can also override the request method:

https://stackoverflow.com/a/32503192/1678094

conn.setRequestProperty("X-HTTP-Method-Override", "PATCH");
conn.setRequestMethod("POST");


Error while trying to run docker redis image on Alpine Linux 3.7.0 Specifications

Microsoft Windows [Version 10.0.14393]
VirtualBox 5.2.8
Alpine Linux Extended 3.7.0

Docker Info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.10.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6facb77383876aeff8a6944dde66b3089574
Kernel Version: 4.9.65-1-hardened
Operating System: Alpine Linux v3.7
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.953GiB
Name: node-4
ID: 3AVF:5SGV:6Y3P:HQW4:PTSU:BRA7:R3PL:AW4F:CIFW:BYDY:FY5P:O6GX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Issue Description

I'm trying to run the bitname-redis-docker image on my Alpine Linux setup on VirtualBox.

docker run --name redis -e ALLOW_EMPTY_PASSWORD=yes bitnami/redis:latest

The container never starts. Inspecting the log it says

Welcome to the Bitnami redis container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis/issues
Send us your feedback at containers@bitnami.com

PROT_EXEC|PROT_WRITE failed.
INFO  ==> Starting redis...
sed: can't read /opt/bitnami/redis/conf/redis.conf: No such file or directory
14:C 26 Apr 19:38:43.389 # Fatal error, can't open config file '/opt/bitnami/redis/conf/redis.conf'

To further investigate I ran an strace and here is the log output link.

Can someone tell me what's going on here? Am I doing something wrong?

One could also use a docker image from an official docker repository like https://hub.docker.com/r/library/redis/

docker run --name some-redis -d redis:alpine


Minimal CMake Jenkinsfile failing Why is this Jenkins build failing? How can I write a Jenkins file just to test the building of this simple project?

DisplayImage.cpp

#include <stdio.h>
#include <opencv2/opencv.hpp>

using namespace cv;

int main(int argc, char** argv )
{
    if ( argc != 2 )
    {
        printf("usage: DisplayImage.out <Image_Path>\n");
        return -1;
    }

    Mat image;
    image = imread( argv[1], 1 );

    if ( !image.data )
    {
        printf("No image data \n");
        return -1;
    }
    namedWindow("Display Image", WINDOW_AUTOSIZE );
    imshow("Display Image", image);

    waitKey(0);

    return 0;
}

CMakeLists.txt

cmake_minimum_required(VERSION 2.8)
project( DisplayImage )
find_package( OpenCV REQUIRED )
add_executable( DisplayImage DisplayImage.cpp )
target_link_libraries( DisplayImage ${OpenCV_LIBS} )

Jenkinsfile

pipeline {
    agent any
    stages {
        stage('build') {
            steps {
                cmakeBuild
                installation: 'InSearchPath'
            }
        }
    }
}

Output from Jenkins

WorkflowScript: 6: Expected a step @ line 6, column 17.
                   cmakeBuild
                   ^

WorkflowScript: 7: Expected a step @ line 7, column 31.
                   installation: 'InSearchPath'

Verify that you have the correct CMake Plugin installed. Also, add parenthesis to your cmakeBuild statement in your pipeline

pipeline {
  agent any
  stages {
    stage('build') {
      steps {
          cmakeBuild(
            installation: 'InSearchPath'
          )
      }
    }
  }
}


Cloud service for Django app deployment We are having webapp which is use Django and Mysql which run in Windows using Apache. Now we are planing to move into cloud service like

1) Google cloud

2) AWS

3) Azure

So to deploy existing Django and move MySQL into database into cloud  which one is easy tool from above 3 list. Or what is the approach that I can follow move my existing app into cloud

The “easiest” approaches to deploying such app is probably using a Platform as a Service (PaaS). Here are some of them to get you started, pick any, really:

Python Anywhere (https://www.pythonanywhere.com/), focused on Python apps (hence Django as well)
Heroku (https://heroku.com/)
Clever-Cloud (https://www.clever-cloud.com/)
Google App Engine (https://cloud.google.com/appengine)
AWS ElasticBeanstalk (https://aws.amazon.com/fr/elasticbeanstalk/)
Azure AppServices (https://azure.microsoft.com/services/app-service/)
RedHat OpenShift (https://www.openshift.com/), open-source, available self-hosted or as a service

To me the first three are the easiest, but it all eventually boils down to personal preference.


Maven dependencies that are downloaded not available in JFrog Artifactory server I have the following repositories structure created in my newly installed Artifactory instance. And we are using this artifactory instance to store the Maven artifacts. Most of our projects are in Java and we use Maven as the build tool. As this is a new installation our .m2 repository cache is empty and when we did our initial 'mvn compile' it has downloaded all the dependencies from Maven Central repository and placed the dependencies under .m2 cache but not in our artifactory instance.

According to our understanding, we thought if the dependencies are not available in Artifactory server then it downloads from a public repository and then it keeps a copy in Artifactory server and also in a local .m2 cache. Is that how it works? When the dependencies are downloaded from public repository we also wanted to have that available in our artifactory server so that when a Maven is installed in a new machine then it should get that dependency from our Artifactory server itself. Is that possible?

Here's the list of our repositories:

Local
 - wellows
Remote
 - wellows-remote
Virtual
 - wellows-virtual

In order Artifactory to cache your dependencies, you need to retrieve them from Artifactory. Once you use your virtual repository, which contains the Maven Central remote, Artifactory will download the dependencies from Maven Central and will cache them for the future use.

So, here's what you need to do in order to populate Artifactory with your dependencies:

Make sure that your virtual repository contains Maven Central or JCenter remotes.
Configure your Maven to work with Artifactory (The "Set Me Up" button in the Artifactory UI)
Delete the ~/.m2/repositories cache
Run your build again.


Trying to make job working AWS MediaConvert (ERROR 1040) Here is my json job.

{
  "Queue": "arn:aws:mediaconvert:xxxxx",
  "UserMetadata": {},
  "Role": "xxxxx:role/MediaConvert",
  "Settings": {
    "TimecodeConfig": {
      "Source": "EMBEDDED"
    },
    "OutputGroups": [
      {
        "CustomName": "123",
        "Name": "Apple HLS",
        "Outputs": [
          {
            "Preset": "System-Ott_Hls_Ts_Avc_Aac_16x9_1280x720p_30Hz_5.0Mbps",
            "OutputSettings": {
              "HlsSettings": {
                "SegmentModifier": "1"
              }
            },
            "NameModifier": "test video out"
          }
        ],
        "OutputGroupSettings": {
          "Type": "HLS_GROUP_SETTINGS",
          "HlsGroupSettings": {
            "ManifestDurationFormat": "INTEGER",
            "SegmentLength": 10,
            "TimedMetadataId3Period": 10,
            "CaptionLanguageSetting": "OMIT",
            "Destination": "s3://xxxxxx/",
            "TimedMetadataId3Frame": "PRIV",
            "CodecSpecification": "RFC_4281",
            "OutputSelection": "MANIFESTS_AND_SEGMENTS",
            "ProgramDateTimePeriod": 600,
            "MinSegmentLength": 5,
            "MinFinalSegmentLength": 1,
            "DirectoryStructure": "SINGLE_DIRECTORY",
            "ProgramDateTime": "EXCLUDE",
            "SegmentControl": "SINGLE_FILE",
            "ManifestCompression": "NONE",
            "ClientCache": "ENABLED",
            "StreamInfResolution": "INCLUDE"
          }
        }
      }
    ],
    "AdAvailOffset": 0,
    "Inputs": [
      {
        "VideoSelector": {
          "ColorSpace": "FOLLOW"
        },
        "FilterEnable": "AUTO",
        "PsiControl": "USE_PSI",
        "FilterStrength": 0,
        "DeblockFilter": "DISABLED",
        "DenoiseFilter": "DISABLED",
        "TimecodeSource": "EMBEDDED",
        "FileInput": "s3://xxxxx/test_video.mp4"
      }
    ]
  }
}

And it falls with :

  Invalid selector_sequence_id [0] specified for audio_description [1]

What i'm doing wrong?

I got this exact same error. I had to add the Audio Selectors. See the following JSON example:

"Inputs": [
  {
    "AudioSelectors": {
      "Audio Selector 1": {
        "Offset": 0,
        "DefaultSelection": "DEFAULT",
        "SelectorType": "LANGUAGE_CODE",
        "ProgramSelection": 1,
        "LanguageCode": "ENM"
      }
    },

If you are using the Java SDK, add something similar to the following to your inputs:

AudioSelector audioSelector = new AudioSelector()
            .withOffset(0)
            .withDefaultSelection("DEFAULT")
            .withLanguageCode("ENM")
            .withSelectorType("LANGUAGE_CODE")
            .withProgramSelection(1);

audioSelectors.put("Audio Selector 1", audioSelector);


How can I ensure consistency between new microservices? My organisation is experiencing an explosion of microservices. We currently have no formalised way of bootstrapping new projects. I'm finding that a team will come to me with a bug in their deployment or build process, and I will spend time on it only to realise I've already resolved it in another project. There's also a lot of inconsistency between projects that I'd like to see standardised.

The changes often involve a single file (e.g. serverless.yml or a Makefile) so a solution involving shared libraries e.g. git submodules doesn't seem viable. Each project will have its own set of configuration that needs to be maintained, e.g. Dockerfiles or serverless.yml, so centralised configuration management solutions for VMs aren't really applicable.

How can I ensure that new microservices conform to organisation standards and include bugfixes/features from existing projects in a way that's easy and intuitive for developers who want to start new projects? What are some best practises around resolving these issues?

The current workflow we have is to ask the person next to you "what project should I clone from to use as a template?" and then delete all the stuff that's not necessary for that project.

I'll add an answer of what my solution is so far, but I'm still really interested in hearing how other organisations are solving this problem and the best practices they have.

To resolve the issue of not having a consistent base to create projects for, my idea is to create a repository (repositories?) of boilerplates/templates and use cookiecutter as a tool to scaffold out new microservices. This way, each project is created from a standard base with all the lessons we've learned as an organisation integrated in to it. Any changes we make can be integrated upstream in to the boilerplate repository. I imagine we'll have templates for Nodejs Docker images, Serverless SPAs, Python lambdas, etc.

To resolve the problem of changes made to the templates being propogated downstream to every project, my solution is to implement a process where owners of microservices are made aware of changes to the template and are then responsible for propagating those changes to their microservice.


Saltstack howto create a Concatenation CA? I would like to have the following a bit more clean. I'm trying to create concatenated CA file using Salt.
To accomplish this I first have to collect all glusterfs.pem files from all my minions and transfer them to the salt-master's minions cache and then output all glusterfs.pem into a single CA file (glusterfs.ca) on the salt-master to later on enroll them again on all minions.
Actually my solution is working but it is simply out of the salt context and looks crappy to me. I'm new to salt, I never worked with reactors, so and I was hoping to find some help (preferably  with an example if possible) on this site.
This is my current situation only using states:
glusterfs/init.sls (only runs on minions)
/etc/ssl/glusterfs.key:
  x509.private_key_managed:
    - order: 9
    - bits: 2048
    - name: /etc/ssl/glusterfs.key

/etc/ssl/glusterfs.pem:
  x509.certificate_managed:
    - order: 10
    - signing_private_key: /etc/ssl/glusterfs.key
    - CN: {{ grains.id }}
    - C: UA
    - ST: Unavailable
    - L: Unavailable
    - basicConstraints: "critical CA:true"
    - keyUsage: "critical cRLSign, keyCertSign"
    - subjectKeyIdentifier: hash
    - authorityKeyIdentifier: keyid,issuer:always
    - days_valid: 3650
    - days_remaining: 0

/etc/ssl/glusterfs.ca:
  file:
    - order: 11
    - managed
    - source: salt://glusterfs/glusterfs.ca

ca.sls (only runs on salt-master):
collect_glusterfs_certs:
  cmd.run:
    - order: 1
    - name: salt -C 'not salt' cp.push /etc/ssl/glusterfs.pem

concatenation_glusterfs_certs:
  cmd.run:
    - order: 2
    - onlyif: 'test ! -e /var/cache/salt/master/minions/{master*,worker*}/files/etc/ssl/glusterfs.pem && test ! -e /srv/salt/glusterfs/glusterfs.ca'
    - name: 'cat /var/cache/salt/master/minions/*/files/etc/ssl/glusterfs.pem > /srv/salt/glusterfs/glusterfs.ca'

I basically first pull all glusterfs.pem to the local salt-master's minion cache and afterwards create the concatenated glusterfs.ca file which I  later deploy onto all the minions I previously collected the glusterfs.pem file from...
Thanks in advance

It feels like what you have done here is implement a discount/bastardized CA (Certificate Authority). I fear that in doing so, you might have inadvertently created a security threat for yourself.
In a traditional PKI deployment, your client would generate a private key and a Certificate Signing Request (CSR) that is then submitted to your Certificate Authority for signing. Your CA would then sign your key after validating the applicant by some means and issue a Certificate to be installed on the client.
What you may have just done by concatenating every single certificate of each client into a CA is grant each and every client the ability to sign certificates for other machines. Any machine in your Gluster cluster then may have the ability to issue a certificate for an arbitrary key which then would be trusted by your Gluster cluster which could allow for an attacker to use a compromised system or key to allow an arbitrary endpoint to join your cluster.
Instead, you might want to follow this guide on setting up a CA for use by your SaltStack. In fact, it is worth noting that your salt master is itself a CA. You could simply copy the minion's certificate and key for use by gluster, though that has the downside of a compromised cert/key being able to be used to impersonate a minion.
But generally speaking, you probably don't want to grant all of your minions the power to be a CA.


Is there an ansible module that check whether a port is not in LISTEN state? The Ansible module index has been consulted, but no module was found that checks whether a port is available? Is there a module that supports this?

What does available mean?

Available means that a port, e.g. 80 is not in LISTEN state. The following indicates that port 111 is unavailable.

user@localhost ~ $ ss -nat
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port
LISTEN     0      128       *:111                   *:*

What is the aim of this question?

The aim of this question is to find a way to stop an Ansible run when port 80 is already allocated by another process than certbot. Certbot requires port 80 to be available, otherwise the cert cannot be renewed. The issue can be found here.

I think you're looking for the wait_for module. It will allow you to check and make an action when a certain port is available and more.


How to automate Powershell's interactive Get-Credential method? I am trying to write a python script which automates the powershell's Get-Credential method. Here is something i wrote:

import subprocess
COMMAND_LINE = 'powershell'
powershell = subprocess.Popen(COMMAND_LINE, shell=True,
                                   stdin=subprocess.PIPE,
                                   stderr=subprocess.STDOUT,
                                   stdout=subprocess.PIPE)
output = powershell.stdout.readline()
print output.rstrip()
powershell.stdin.write('$cred = Get-Credential\n')
powershell.stdin.write('Administrator\n')
powershell.stdin.write('Password')
output = powershell.stdout.readline()
print output.rstrip()
out = powershell.communicate()[0]
print(out)

But i am unable to pass the password as above, i am getting following exception:

> PS /root> $cred = Get-Credential
>
> Windows PowerShell credential request Enter your credentials. User:
> Administrator Password for user Administrator: PS /root>
> PasswordPassword : The term 'Password' is not recognized as the name
> of a cmdlet, function, script file, or operable program. Check the
> spelling of the name, or if a path was included, verify that the path
> is correct and try again. At line:1 char:1
> + Password
> + ~~~~~~~~
>     + CategoryInfo          : ObjectNotFound: (Password:String) [], CommandNot    FoundException
>     + FullyQualifiedErrorId : CommandNotFoundException

As i am new to python, Can anyone help me here?

Your python problem is the \n after the commands, write already append a newline, so your script does the following:

$cred = Get-Credential # Start the command
# This line is ignored as extraneous after Get-Credential, but you can see it in your output.
Administrator # Send proprely to the User: prompt
# Sent to the Password: prompt
Password # Send to the shell input, causing the error at end of your log.

Now that's a quite hard way to automate the credential object in powershell, usually the object is build by other way like described in this blog post (quoting here for completeness):

$password = "mypassword" | ConvertTo-SecureString -asPlainText -Force
$username = "nwtraders\administrator"
$credential = New-ObjectSystem.Management.Automation.PSCredential($username,$password)

And used as such:

Get-WMIObject win32_logicaldisk -ComputerName Server1 -Credential $credential


Amazon EKS: using spot instances for worker nodes When using AWS EKS, is it possible to set up the worker nodes on spot instances?
how?
Anything I special I should pay attention to, in such a setup?

It can be done by modifying the Cloudformation Template (which is mentioned in this document) in the LaunchConfiguration section to specify a spot price.

NodeLaunchConfig:
  Type: AWS::AutoScaling::LaunchConfiguration
  Properties:
    SpotPrice: "20" # <=== Here
    AssociatePublicIpAddress: 'true'
    IamInstanceProfile: !Ref NodeInstanceProfile
    ImageId: !Ref NodeImageId
    InstanceType: !Ref NodeInstanceType
    KeyName: !Ref KeyName
    SecurityGroups:
    - !Ref NodeSecurityGroup
    BlockDeviceMappings:
      - DeviceName: /dev/xvda
        Ebs:
          VolumeSize: !Ref NodeVolumeSize
          VolumeType: gp2
          DeleteOnTermination: true
    UserData:
      Fn::Base64:
        !Sub |
          #!/bin/bash
          set -o xtrace
          /etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
          /opt/aws/bin/cfn-signal --exit-code $? \
                   --stack  ${AWS::StackName} \
                   --resource NodeGroup  \
                   --region ${AWS::Region}


What does "eksctl create iamserviceaccount" do under the hood on an EKS cluster? AWS supports IAM Roles for Service Accounts (IRSA) that allows cluster operators to map AWS IAM Roles to Kubernetes Service Accounts.
To do so, one has to create an iamserviceaccount in an EKS cluster:
eksctl create iamserviceaccount \
    --name <AUTOSCALER_NAME> \
    --namespace kube-system \
    --cluster <CLUSTER_NAME> \
    --attach-policy-arn <POLICY_ARN> \
    --approve \
    --override-existing-serviceaccounts

The problem is that I don't want to use the above eksctl command because I want to declare my infrastructure using terraform.
Does eksctl command do anything other than creating a service account? If it only creates a service account, what is the YAML representation of it?

Yes It does more than just creating a service account. It does three things:

It Creates an IAM role.
It attaches the desired iam-policy (--attach-policy-arn
<POLICY_ARN>) to the created IAM role.
It creates a new kubernetes service account annotated with the arn of the created IAM role.

Now It's easy to declare the above steps using kubernetes and aws providers in terraform.


Override Ansible task parameters from role I'd like to be able to define a task with default parameters:

- name: Create a new user
  user:
    name: "default"
    password: "password"
    state: present

Then, from my playbook, I'd like to be able to do something like this:

  roles:
    - role: common
      vars:
        name: "frank"
        shell: "/bin/zsh"

...and have it create the user "frank", with the password "password" and the /bin/zsh shell. Is it possible to override task parameters from an Ansible playbook? (For the record, the above doesn't work, I tried it.)

Defining default value for ansible variables and overriding them via ansible playbooks can be achieved using Jinja2 templating.

- name: Create a new user
  user:
    name: "{{ username_variable | default('default_value') }}"
    password: "{{ password_variable | default('default_value') }}"
    state: present

Then if these variables are passed from the playbook it will be used, or else the default values.

  roles:
    - role: common
      vars:
        username_variable: "frank"
        password_variable: "/bib/zsh"


Jenkins: Re-use repository with reference option and do changes on it I would like to setup a Jenkins cluster where the slaves are building an Android ROM with some parameters like branch (e.g. lineage-16.0) and some optional patches. The whole repository will consume around 100GB.
Let's say i want to build branch lineage-16.0 with patch-A and patch-B and in parallel build branch lineage-16.0 with patch-A only and in parallel build branch lineage-17.1 with no patches and so on.
I just read about the --reference option in git (which can also be used in repo).
Is it possible to use it this way or will changing the branch and applying patches modify the reference clone as well as it only uses pointers?
I need to save disk space and network traffic somehow.
If this is not possible is there any other way without duplicating the 100GB for each jenkins job?
I'm also open for other free tools than Jenkins.

If this is not possible is there any other way without duplicating the 100GB for each jenkins job?

To avoid frequent clones of the repository, you can put a single clone of the repository on-disk on your executor nodes outside of the normal workspace path.  Then, in your Jenkins job, you can cd to that directory with the dir step, update the repo, and checkout the branch you want, e.g.:
dir('/path/to/repo') {
  sh('git fetch --all')
  sh('git checkout origin/lineage-16.0')
  // your build steps here
}

However, this has the major drawback that the one repo will be shared across all of your jobs, which means you cannot run those jobs in parallel without risking corruption.  It sounds like running jobs in parallel was one of your requirements, so this may be a no-go.
You could probably get parallel builds to work by using a copy-on-write filesystem so that the repo appears to be in multiple independent directories but only actually has to be cloned once, but I've never tried something like this, so I can't say how well it would work.


ansible: difference between a variable and a fact While I'm using Ansible for quite some time, I'm not sure I really understand the differences between a variable and a fact.
Would it be any difference doing
- set_fact:
    nginx_ssl: /etc/nginx/ssl
    nginx_conf_file: /etc/nginx/nginx.conf

or
  vars:
    nginx_ssl: /etc/nginx/ssl
    nginx_conf_file: /etc/nginx/nginx.conf

Could someone explain me, possibly with an example where it really make a difference ?

Ansible facts are data collected about the (target) systems on which Ansible takes actions. They are variables, but set by Ansible (in a way like system defined variables). They are collected during Gathering Facts stage of a playbook run, and it is controlled by the gather_facts setting. Ansible calls this variables discovered from systems. That said, it is possible to set custom facts also.
Some examples are:

ansible_hostname - the FQDN of the target system
ansible_os_family - the Operating System family of target system (RedHat, Debian, etc.)

The other variables are the ones we can set as per our requirement (in a way like user defined variables).
Some examples are:

my_fav_fruits: [ 'orange', 'apple', 'banana' ] - yours might differ.
config_dir: '/etc/my_app/conf.d' - for my application configuration files.

Update:
Updating answer to make it relevant to the edit(s) made in question.
As @Bruce Becker's answer rightly pointed out, there is a difference in the precedence of variables set with set_fact. Also variables that need to be set at "run time" can be set this way. Without further explanation, taking your example variables, if I create a play like below:
- hosts: localhost
  vars:
    nginx_ssl: '/etc/nginx/ssl'
    nginx_conf_file: '/etc/nginx/nginx.conf'

  tasks:
  - name: set nginx path to /opt when running on Debian
    set_fact:
      nginx_ssl: '/opt/nginx/ssl'
      nginx_conf_file: '/opt/nginx/nginx.conf'
    when: ansible_distribution == 'Debian'
  - debug:
      msg: 'ssl: {{ nginx_ssl }} and conf: {{ nginx_conf_file }}'

Then the set_fact variables will take precedence (on Debian) and output will be:
"msg": "ssl: /opt/nginx/ssl and conf: /opt/nginx/nginx.conf"

On other distributions, it will be what was declared in vars:.


Sending Exceptions to Elastic Stack via REST I am working with devs wishing to replace Exceptionless, a Real-time exception reporting tool.

They wish to send the exceptions from a web app that they would previously send to Exceptionless to our ELK stack.

I am only experienced with using Beats to ship data to ELK.

In the past I have written custom code to do similar things like this, but we shipped to Splunk, not ELK.

How friendly is the process of sending information to ELK via REST? Are their tips, tricks and/or pitfalls?

Is this a supported, recommended, and good way to do exception logging/reporting/analytics/etc.

It depends on what you're trying to accomplish but IMO if you're interested in reliably getting your exceptions you probably want something between Elasticsearch and your app - a queueing or logging system of some kind like Kafka or Redis.
Send your exceptions there and then drain that to Elasticsearch. Otherwise you run the risk of overwhelming ES's ability to accept writes at the very time you need it most (cascading failure, DDOS etc).


How can I expose Ingress service on 80 and 443 to the outer world I used google compute engine to setup a Self Managed Kubernetes, While exposing ingress service via NodePort/LoadBalancer/EnternalIP, it is using default 30000 port range, But I want to expose it on 80 and 443 to the outer world. Please suggest How can I do that?

It's would be better if you will use ClusterIP type as a service than apply Ingress resource on it
https://kubernetes.io/docs/concepts/services-networking/ingress/#single-service-ingress


Why use EXPOSE in Dockerfile -- since you can bind to all ports anyways I can docker run -p 3000:3000 image without EXPOSEing that port in the container (see below). If that's true, then why bother putting EXPOSE in the Dockerfile? Is it just for communication to image users? Because I don't know of a functional reason to EXPOSE ports if they are all bindable anyways.

Here are the steps showing me binding to a port in a container despite the fact it is not EXPOSEd

$ cat Dockerfile
FROM alpine
RUN apk add nodejs npm vim
COPY webserver /webserver
CMD [ "node", "/webserver/index.js" ]

$ docker build .
Sending build context to Docker daemon  1.931MB
Step 1/4 : FROM alpine
 ---> 11cd0b38bc3c
Step 2/4 : RUN apk add nodejs npm vim
 ---> Using cache
 ---> 4270f8bdb201
Step 3/4 : COPY webserver /webserver
 ---> Using cache
 ---> 67f4cda61ff0
Step 4/4 : CMD [ "node", "/webserver/index.js" ]
 ---> Using cache
 ---> 1df8f9024b85
Successfully built 1df8f9024b85

$ curl localhost:4400
curl: (7) Failed to connect to localhost port 4400: Connection refused

$ docker run -d -p 4400:3000 1df8f9024b85
7d0e6c56f8ad8827fe72830a30c1aac96821104b8ea111291ca39e6536aad8fd

$ curl localhost:4400
Hello World!

$

Docker's EXPOSE documentation addresses this specific point:

  The EXPOSE instruction does not actually publish the port. It
  functions as a type of documentation between the person who builds the
  image and the person who runs the container, about which ports are
  intended to be published. To actually publish the port when running
  the container, use the -p flag on docker run to publish and map
  one or more ports, or the -P flag to publish all exposed ports and
  map them to high-order ports.

Pay attention to the last sentence, if you expose multiple ports then -P becomes useful to avoid setting multiple -p on the command line.


Is Windows in general always slower than linux? There are multiple distribution in the company. Building the same project results in different build times (see table 1).

| win10laptop32gbram | win10laptopOld | win7laptop | win7server | linuxvm | mac |
|--------------------|----------------|------------|------------|---------|-----|
| 3                  | 3              | 2          | 1.5        | 1       | 1.9 |

Table 1: Ratio of build speed between different Operating System and hardware.

Do other people from other organizations experience this as well?

TL;DR depends on the use case with up to 12 times between best/worst

Comprehensive testing has been done over at Phoronix (28.03.18) and the results are very mixed. The tests looked at:

Clear Linux 21510
Debian 9.4
OpenSUSE 42.3 Leap
Ubuntu 16.04 LTS
Windows 10 Pro Build 16299
Debian 9 On WSL
OpenSUSE 42.3 On WSL
Ubuntu 16.04 LTS On WSL

The results are all over the show. In a few tests, the WSL (Windows Subsystem for Linux) is faster then native Windows (GraphicsMagick) and the Intel optimized distribution, Clear Linux, sometimes has the worst results (Netperf UDP request response). While in others Windows is twice as fast as all Linux distributions (CacheBench). All in all, deciding on performance is not conclusive. Choose the OS that provides the tooling you require.

Nonetheless, the article itself does provide a winner given that only first place finishes are taken into account:

  ... Clear Linux was the fastest of the operating systems tested with
  coming in first 40% of the time, Debian 9.4 in second with first place
  finishes 22.5% of the time, and Windows 10 Pro itself was the fastest
  12.5% of the time.

When only taking Java into account:

  In the Java tests, to no real surprise, the performance tended to be
  about the same across all tested operating systems.

And finally, in a compile test (Golang), which is the test that is most related to the use case in the question:

  The build performance on Windows 10 and WSL were slower than the bare
  metal Linux performance.


Kubelet config file -- > clusterDNS clusterDNS:

  - "10.32.0.10"

The above is a flag passed to the kubelet file, i dont know what it refers to clsuter DNS ? Is it refers to VPC CIDR or any subnet CIDR ?

It is the address of the DNS server (probably CoreDNS but possibly kube-dns) for the cluster.

Kubernetes uses a DNS server within the cluster so pods can find each other using service names. This is the "cluster DNS" server. Every time a service is created, it gets registered in the DNS server.

In Linux, the /etc/resolv.conf file is where the DNS server is configured. If you want to use Google's public DNS server (8.8.8.8) your /etc/resolv.conf file has the entry nameserver 8.8.8.8.

In Kubernetes, the kubelet process on a worker node configures each pod.  Part of the configuration process is to create the file /etc/resolv.conf and specify the correct DNS server.  That server is spec'ed by the clusterDNS configuration option.

How you set it (if you even need to set it yourself) depends on how you're installing kubernetes.  It ultimately depends on the CNI provider you're using, but also if you're doing "kubernetes the hard way" versus kubeadm.


What is the major difference between continuous delivery and continuous deployment I am a beginner in devops.I am getting confused with the answers on google so posted here.

"Continous delivery" is the general principle of a development process (including development, testing, etc.) producing software in a way where you could deploy often, ideally at any time. In contrast to a classical waterfall model where you would have fixed deployment dates which would be months or even years apart and planned with huge feature sets. These days, this means that there is some CI/CD pipeline which may or may not include the final step of "deployment".

"Continuous deployment" is a specific aspect which means that the final step of the process, the actual deployment, is automated as well, and happens very frequently, ideally on every commit (after running through a rigorous CI/CD pipeline which may or may not include any human interaction).

There is no versus here. Continuous Deployment is an optional sub-part of Continous Delivery.


Get all change logs of since last successful build in Jenkins Pipeline In my Jenkins pipeline, I can get change logs of the current build by this.

def changeLogSets = currentBuild.changeSets

Is there a way to get all change-logs since last successful build?

You may try something like this:

node('my_kubernetes_pod') {
    passedBuilds = []

    lastSuccessfulBuild(passedBuilds, currentBuild);

    def changeLog = getChangeLog(passedBuilds)
    echo "changeLog ${changeLog}"
}

def lastSuccessfulBuild(passedBuilds, build) {
    if ((build != null) && (build.result != 'SUCCESS')) {
        passedBuilds.add(build)
        lastSuccessfulBuild(passedBuilds, build.getPreviousBuild())
    }
}

@NonCPS
def getChangeLog(passedBuilds) {
    def log = ""
    for (int x = 0; x < passedBuilds.size(); x++) {
        def currentBuild = passedBuilds[x];
        def changeLogSets = currentBuild.rawBuild.changeSets
        for (int i = 0; i < changeLogSets.size(); i++) {
            def entries = changeLogSets[i].items
            for (int j = 0; j < entries.length; j++) {
                def entry = entries[j]
                log += "* ${entry.msg} by ${entry.author} \n"
            }
        }
    }
    return log;
}

Additionally, you may try out "Changes Since Last Success Plugin":
https://wiki.jenkins.io/display/JENKINS/Changes+Since+Last+Success+Plugin

But it will not work together with pipeline, this is a separate approach.


Failed to install kubernetes on centos Following the official documentation, I have an error when trying to install kubernetes cluster yum package.
I am on Centos 7 using yum. So I run this commands:
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

# Set SELinux in permissive mode (effectively disabling it)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Indeed when I am trying to run yum install -y kubelet kubeadm kubectl I have this error:
Loading mirror speeds from cached hostfile
No package kubelet available.
No package kubeadm available.
No package kubectl available.
Error: Nothing to do

What if the real problem about the installation ?

When I navigate to the URL, there do not appear to be any packages in the repo. Consider trying a mirror like: mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64


Weavenet for kubernetes is showing 0/2 I'm very new to kubernetes. I thought i would first get a basic by making hands dirty by "the hard way"
I followed the Linux academy, but at the networking side using weave net i am now stuck !
completely stuck !

kubectl get pods -n kube-system  -->getting the following crash loop back

weave-net-zngp6   1/2       Error              15         37m    ````

some logs for investigating
````kubectl -n kube-system logs weave-net-px6z7 -c weave-npc````

got
```` Get https://10.32.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.32.0.1:443: i/o timeout````

Not sure what it means ? any one can save me ?

weave-net is trying unsuccessfully to talk to the apiserver at 10.32.0.1 (port 443). Verify your master node is indeed at that IP address, it has network connectivity to the node where this is barfing, and that the firewall on the master node has port 443 open.

If they are all running on the same host, verify the apiserver is running.


Is it possible to send email notification for Input (GO/NOGO) from a pipeline There is User input functionality in the declarative pipeline which prompt on the Jenkins console but it's a headache to login every time on console and select the appropriate action.

So I want to implement such a thing via mail for User input, For example User will get the Input via mail where there are two URL's for "Approval" or "Reject" respectively, If User select Approval then proceeded otherwise not. Here need to pass input on the pipeline on runtime.

Are there any plug-ins for same ?

You don't need any plugins.  You can send an e-mail from your Pipeline using the mail() step.  You can respond to the input step by using the REST API as documented in this answer on Stack Overflow.  Put the two together and you should be able to craft an e-mail that contains buttons for responding to the input.


How can I 'aws s3 sync' two buckets, which are located in different accounts I'm trying to use 'aws s3 sync' on the awscli between two accounts.

Account A, I own.
Account B, Owned by a third party.

Account B has given a user:jon on account A permission to a bucket through a role:assumeDevOps assumption.

Jon assumes assumeDevOps to access bucket on Account B. But now I have to sync to a bucket back on account A.

I'm getting an access denied. Possibly because that role that Jon assumed has no permissions to the bucket back on my account.

How do I do this?

Is there documentation on this kind of situation?

Attach a policy to the source bucket

Get the AWS account ID number of the destination account.
From the source account, attach a policy to the source bucket that allows the destination account to get objects, similar to the following:

Important: For the value of Principal, be sure to enter the AWS account ID number of the destination account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "222222222222"},
            "Action": ["s3:ListBucket","s3:GetObject"],
            "Resource": [
                "arn:aws:s3:::sourcebucket/*",
                "arn:aws:s3:::sourcebucket"
            ]
        }
    ]
}

Attach a policy to a user or group in the destination account

Attach a policy to the destination account's IAM user or group that allows the user to copy objects from the source bucket to the destination bucket. The policy can be similar to the following example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::sourcebucket",
                "arn:aws:s3:::sourcebucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::destinationbucket",
                "arn:aws:s3:::destinationbucket/*"
            ]
        }
    ]
}

Copy objects from the source bucket to the destination bucket

After you set up the policies on the source bucket and the destination account, the destination account can copy objects from the source bucket to the destination bucket. Then, the destination account owns the objects copied into the destination bucket.

To synchronize all content from the source bucket to the destination bucket, you can run this command:

aws s3 sync s3://sourcebucket s3://destinationbucket

for more details check this link OR this link


Use a pod template file in a Deployment I've been looking at Deployments descriptions in Kubernetes. In the tutorials I've found, the definition of the replicated pods are always a copy-paste of a pod template file.

Is it possible to rather use a reference to the yaml file (or an url) defining a pod (I didn't found any resource telling so)?

If yes, should it be avoided (in order to keep label/selector in the same file for example)?

If no, is there a reason (because Kubernetes wouldn't know if you modified the pod template file for example)?

Yes you can create resources (including pods) from a url yaml using kubectl.  However, just because you pointed to a remote url does not mean that the resource will automatically update.  You will need to either re-deploy when the remote yaml is updated, develop/integrate with a service that handles this, or look into using a tool such as helm to help manage changes.

It does not need to be avoided, you just need to know that it will not automatically update if the source changes.


Where do I start learning DevOps? Searching "how to start with devops" didn't really help as, many of them like this and this as well as many others don't really point me in the right way. Every youtube video I have seen so far hasn't helped either (as I don't need videos explaining to me what IS devops), this video being the only true exception so far (that shows how things integrate with each other).

I am a CCNA and also highly proficient with DBMS-s (currently learning parallelism execution as well as other stuff).

But I have no idea what I am supposed to learn first (or where to find the correct resources for that matter).

I am about to graduate, so I have no access to real world experience.

Any advice or help?

Devops relate to the execution of software. It is all about how to run an application or service. So server configuration tools, virtual machines and containers are the next steps. I suggest looking at the documentation of those tools and learning from them things like discovery, observability, scalability and redundancy. Just to name some examples:

Server config:

PXE
Ansible
Puppet

VM:

Vagrant
QEMU

Container:

Docker
Kubernetes

EDIT: Cloud technologies overview:

CNCF landscape

When your software architecture is about microservices, then containers are a good choice. When you are using old monolithic applications then VMs are not a bad option. So it always depends on the context what you need to learn specifically.
The topic is rather broad to summarize it. A book might help with that but sadly I cannot suggest a good broad one. "Kubernetes in Action" gives you a good start in Kubernetes though.


Error while following Docker get-started part 3 I am learning Docker by following the get started on its official page.

I got stuck at part 3 when I tried to run

docker service ps getstartedlab_web

the containers fail to start. Checking the logs reveals this

'app.py': [Errno 2] No such file or directory

I saw that this issue has been reported elsewhere and the reason was because of the conflict between the volume in docker-compose.yml and workdir in dockerfile.

But my docker-compose.yml doesn't have that volume property, so what's the problem?

Update
I understand the error message. But I don't know why such an error message was shown. I was able to push/pull and start the container alone with docker run (without docker-compose). The problem only arises when I tried to use docker-compose to scale it.

So what's the problem?

  'app.py': [Errno 2] No such file or directory

I think you already answered your question. The troubleshooting part, you suspect something, but you did not test it to see if it is really what you think.

Create two more files, requirements.txt and app.py, and put them in the same folder with the Dockerfile. This completes our app, which as you can see is quite simple. When the above Dockerfile is built into an image, app.py and requirements.txt is present because of that Dockerfile’s COPY command, and the output from app.py is accessible over HTTP thanks to the EXPOSE command.
Get Started, Part 2: Containers


What are known component intelligence tools? What are available - free and commercial - solutions to create reports about components used in a software project (physically, set of source code repositories), with intelligence on features like security situation and license model of used open source components?

One prominent example is - or was - Nexus Enterprise, but are there other solutions?

An open source monitor that comes to my mind is the Census project for security monitoring:
https://github.com/linuxfoundation/cii-census

Note: my impression about Nexus Enterprise is about 2 years old, since then the product name and scope has changed afaik and there are also competitors like Seerene.

(upon request from comments)

BlackDuck has tools to monitor use of open source components, and "manage open source compliance & security"


How to "docker-compose build" a service from code located in local git repository? I have a git repository placed in local file system at /home/nou/code/lib.git
Also I have /home/nou/docker-compose.yaml that is used to deploy a service based on the code from the repo:

version: '2'
services:
  lib:
    container_name: lib
    build: git://./code/lib.git/
    #build: git://./code/lib.git
    #build: git:///home/nou/code/lib.git/
    #build: git://file:///home/nou/code/lib.git/

As you can see, am trying to build lib service from the local git repo, but the only result I observe is such error (or similar):

nou@ubuntu:~/$ docker-compose build
Building lib
ERROR: error fetching: fatal: unable to connect to .:
.: No address associated with hostname

: exit status 128

How can I use local git repo to build services using docker-compose?

What you are specifically after (building from a bare repository on a local filesystem) isn't functionality offered by Docker, and by extension, docker-compose. Docker supports building from a few different URLs, but not bare repositories on local filesystems. You can try a few workarounds:

Build straight from Github: docker build https://github.com/docker/rootfs.git
Run a Git daemon to use the git:// protocol: git daemon --base-path=. --export-all (in /home/nou/code)
Running a Git daemon from within Docker, mounting your bare repository as a volume and building inside it via a mounted Docker socket
Converting your bare repository to a normal one

https://docs.docker.com/engine/reference/commandline/build/#build-with-url


cannot import OpenCV in a Docker container in Raspberry Pi I am trying to build a Docker container in Raspbian 10.

FROM python:3.5-buster
RUN apt-get update
RUN apt-get install apt-utils -y
RUN apt-get install python-opencv -y
RUN apt-get install python3-opencv -y
COPY a.py /
CMD ["python3", "a.py"]

a.py is simply import cv2

When I run the container, I got the following errors.

Traceback (most recent call last):
  File "a.py", line 1, in <module>
    import cv2
ImportError: No module named 'cv2'

Seems to be just an issue with a Python package.

Traceback (most recent call last):
  File "a.py", line 1, in <module>
    import cv2
ImportError: No module named 'cv2'

Is a common Python error, basically it can't find the package cv2.

You could try and install OpenCV through pip instead of installing through apt get.
Something could be up with dependancies that you may have missed.

RUN pip3 install opencv-python


Availability/Reproducibility of Ubuntu Minimal as Docker image While for Debian you find on Docker Hub "slim" distributions, it seems not the case with Ubuntu.

Now there is a "minimal Ubuntu for public clouds" distro from Canonical, but I can find just cloud VM images; is there a Dockerfile for that, as well?

The answer to the question "is there a Dockerfile for that, as well?"

https://github.com/tianon/docker-brew-ubuntu-core/tree/dist-amd64

Note their git branches contain the actual release build based on architecture.


Is it possible to build docker images using docker-compose concurrently? When one runs docker-compose up --build then the docker images are created sequential. Is it possible to build these images concurrently using docker-compose?

There are multiple issues including this, but it is unclear whether it is possible or not.

Yes, it's possible. You can use --parallel option since docker-composer 1.23.2 version.

Version 1.23.2 also correct 1.23.0 bug:

  Reverted a 1.23.0 change that appended random strings to container
  names created by docker-compose up, causing addressability issues.
  Note: Containers created by docker-compose run will continue to use
  randomly generated names to avoid collisions during parallel runs.

https://github.com/docker/compose/releases/tag/1.23.2

You can use it simply doing:

docker-compose build --parallel


Docker swarm stop spin up containers at 250 TLDR: Docker won't spin up more then 250 containers.

I'm deploying a cluster of 3 docker services to a swarm with 2 nodes. 2 of the services need to contain 1 container (have a replicas: 1 in the docker-compose file), and the third service need to have 300 containers (have a replicas: 300 setting).

The problem is it's spin up those 3 services, the first two with 1 container each (work like they should), and the third service spin up 248 containers out of 300 (I see this when I do docker service ls). I try to search if there is a limit of the service or swarm but couldn't find any.

I will much appreciate any help I can get.

If it's matter, each node with a 30GB RAM and 8 cores, and I use only 1/3 of the RAM.

I just figure it out. The problem is not with the service or the swarm, it's with the network.

When I use driver: overlay the default subnet is 10.0.0.0/24 which result in 254 address. So I change the mask in the subnet, to 22, which result in 1022 address, I added:

ipam:
  config:
    -subnet: 10.0.0.0/22

And now the network section in the docker-compose file looks like this:

networks:
  web:
    driver: overlay
    ipam:
      config:
        - subnet: 10.0.0.0/22


windows 10 vs windows server 2016 as jenkins slave The builds are running locally in my Windows 10 OS successfully. Can I run the same builds in Jenkins slave with Windows Server 2016? Will there be any issues during the build or should I go for Windows 10 OS for the slave setup?

Unless you do something distro-specific (i.e. something which depends on the specific version of Windows) there should be no issues.


What is an artifact repository? What is an artifact repository?  If it's just a place to store files, can't I just use a source control system?

During development you generate a fair amount of different artifacts. These might include:

The source code
The compiled application
A deployable package
Documentation

and potentially others as well

While you could use a source control system to store all of them, it's usually massively inefficient, as source control systems are usually designed to handle text based files, and not binary files. You might be able to use them as a simple storage mechanism, if most of your releases are text based, and you don't have to store a lot of binary data.

Artifact repositories however are designed to store all kinds of files, including binary ones. This includes anything from zipped up source codes, to build results, to things like docker images as well. Also, they usually not only store these artifacts but also help manage them using various additional functions, for example:

Versioning support: properly store some metadata, like when each artifact was built, what their version number is, store their hashes, etc.
Retention: make sure you only keep the important artifacts, and automatically delete ones that are only snapshots / not needed anymore, etc. based on various criteria you can set up
Access control: set up who can publish and who can download the various artifacts
Promotion: ability to promote artifacts. For example you can have snapshot artifacts with a short retention period on a server near your coders, and a separate repository near the live servers, where only artifacts that have been deemed deployable appear. This also includes support for various version channels, and moving artifacts between them (like promoting a specific version from beta to stable).
Act as a native repository for the artifacts. Meaning you can use it as the main repository for maven, rubygems, docker, etc. This can also include caching of artifacts from the official repositories as well.


Reset Gitlab Project to Repository Content I had some unwanted files in a repo which is going open-source soon. I removed the files using git-filter-branch. I noticed the project size in Gitlab growing with every such removal. Some research suggested that Gitlab has another backup layer, where the data is still in the project, which can be accessed by exporting the project.
Is there a way to just reset the Gitlab storage to what the repo actually contains when being cloned?
I'd much prefer not to do these operations again on the project file, since it is prone to data loss.

The solution I found in the end is to just create a blank new project, add that to your git remotes and then push to it.
You will have to manually carry over your Issues and probably some other stuff as well. Note that issues can be exported and imported as well, but they loose their comments in the process.


What's difference between on-premise and SaaS? I just curious what's difference between on-premise and SaaS?

SaaS (Software as a Service), often referred to "on-demand Software", is a model where you're provided with access to the software in the cloud. There's no installation, setup or running of the setup, as the service provider will do this for you. You just have to pay the service provider. Example: Office 365
On-Premise is the "traditional" model where you have to install, setup and run the software at a computer at your physical location. It takes time, personnel and possible equipment. It may also have additional cost to hardware and software. Example: Office 2016

Office 365 is SaaS which provides an online version of the MS Office suite. While Office 2016 is On-Premise Software that will require installation and running.

In addition

PaaS (Platform as a Service) provides a platform which typical includes operation system, databases, web server, containers for executing programming languages. Examples: Windows Azure, Azure SQL, Heroku
IaaS (Infrastructure as a Service), provides as name suggesting infrastructure, like virtual machines, storage, network, network service (firewalls). Examples: Windows Azure, Google Compute Engine


What are benefits of using third party app metrics tools versus custom build tools? With tools like Riemann, New Relic, etc., is there any benefit to building custom app metrics tools for monitoring application health?

A third party app metrics tools is great if it already covers all (or most of) the parameters you want to monitor and it presents the information in a convenient enough manner for your use case - you just need to (find it first and then) learn how to install/use/maintain it and then do that.

Advantages:

avoids the software development and maintenance costs, which can be quite high depending on the capabilities
can be immediately used, a custom tool needs to be developed first
technical support (not always great, tho)
standardisation: if the tool is a popular one - expertise on it can be requested on a job description instead of having to train every user on it

However there are many cases in which custom development is needed, some already mentioned by Michael: reliability, security/confidentiality, lack of integrations for your custom applications.

Depending on each particular case, custom development can go from tiny scripts extending the coverage of a third party tool to complex wrappers assembling multiple such tools to achieve the necessary monitoring capabilities

Such development is not always lightweight:

it may require in-depth tool knowledge and/or language expertise outside of your team
it may need management for heavy/complex or even conflicting dependencies
the adaptation layers for the tool's actual usage environment may be very heavy

Sometimes the cost of such development is higher overall than that of developing a custom system.

Othertimes the functionality actually needed from the third party tools is a relatively small percent of its capability set and the overhead for learning/installing/operating the tool is simply not worthy.

You guessed it - I wrote my own monitoring tool :)


How to get the latest stable version of Firefox? How to get the latest version of firefox? This URL was found, but is there no API? I would like to get the latest version, e.g. 58.0.2 so this could be used for other purposes. Just latest is insufficient as I require the version in one of my ansible roles to lookup the checksum.

At the moment I am considering to parse https://www.mozilla.org/en-US/firefox/releases/

I pulled this from the Firefox Chef cookbook (in particular this .rb file). This an example of the request they use to get the latest version based on OS and language.

https://download.mozilla.org/?product=firefox-latest#&os=win#&lang=#en-US

It adheres to the following format:

https://download.mozilla.org/?product=firefox-{VERSION_NUMBER}#&os={OS_ABBREVIATION}#&lang=#{LANGUAGE_ABBREVIATION}

I searched around for official api documentation, but couldn't find anything. If I do, I will post a link to it in this answer.


What can be considered as an artifact Recently, I have had a lecture at my university about advanced testing and CI. To understand all concepts, I was searching what artifact represents, specifically in DevOps sphere. I found many contradictory statements, definitions, and articles.

Therefore, I would like to ask, if there is a standard (or at least some unwritten rules) of what everything could be considered as an artifact. What are the most important parts of artifact which should always be included?

Many sources say that artifact should not be source code, but deliverable, and JDK is often used as an example. In that case, I would like to know what is the equivalent when working with scripting languages like JavaScript or Python.

Thank you for all your explanations!

Artifact is an abstraction. It could be anything that's produced by somebody else. In Bulgaria for example we're making artifacts for the winter time ( it's called 'compot') . Basically it's an artifact, it was produced by my grandmother and could be used by me :)

Same goes into the IT world, but with one small difference.

The Java artifacts are a little bit more special, since Java is a static language and before the run phase you would have those compiled artifacts - *.jar files. After that you could reference them during the 'run' phase. This is the reason why they are always pointed as the most straightforward artifact example, the java compiler will produce them.

One major distinguishment between my grandmother's artifacts and Java's ones is that it's a good practice to have a version for each artifact, that way you could easily switch them and see what works and what does not.

JavaScript and Python are dynamic languages, they do not have any compiled artifacts like Java, you could just refer to the '.py' file.

As a summary an artifact could be anything:
from tar.gz to a jar file, but it must always be versioned :) Of course the versioning might come during a later stage, like a docker image tag, but the principle still remains.


What are the "User Mode" tools in kubernetes lens? I have been using kuberentes lense and noticed under the cluster settings page an optional package called "user mode".  Does anybody know what this means?

I have not used it, but the description in your image is understandable. It is adding RBAC (user permission) so that any user (also non-admins) can list namespaces (e.g. see all namespaces that exists within the cluster).
But it should not mean more than that, e.g. a user can not create a Deployment in a namespace he is not granted to.


Is it possible to set CloudWatch alarm for a combination of metrics? For example, I want to trigger a CloudWatch alarm for the following:

Number of messages sent - Number of messages deleted >= 6

The above is for AWS SQS metrics. And I want that metric to trigger AWS Lambda if that alarm is set off.

Is there any way I can do it? If not, then is there any work-around [Aiming for a serverless paradigm]

[I want to do that, cause there's no metric which tells me the number of messages currently (available) in the queue].
So, if there's a metric which can give me that, it'd be very helpful

For the last overall goal, I would set the alarm on ApproximateNumberOfMessagesVisible from Cloudwatch documentation on SQS this is:

  The number of messages available for retrieval from the queue.

  Units: Count

  Valid Statistics: Average, Minimum, Maximum, Sum, Data Samples
  (displays as Sample Count in the Amazon SQS console)

For the more generic computed metric question I would go with a lambda running priodically, gathering metrics from cloudwatch, doing the maths and pushing back to cloudwatch.

Here is an example lambda gathering data from Cloudwatch to push to elasticsearch. With the examples on the documentation you should be able to push metrics back from the lambda.

Be aware that calling cloudwatch API to gather and push metrics has a cost when you got over 1M request per month, there's other limits to be aware of to avoid breaking your budget/workflow.


How to implement GitLab Flow branching model for microservices? Recently I’ve been looking into alternatives to the git flow branching model and since I’m already using GitLab, I stumbled upon their own branching model called GitLab flow. I’m currently trying to get my head around how to implement this properly for microservices. We will assume, that I am deploying all services to the same kubernetes cluster, each deployment should reside in its own namespace.

If I understand this model correctly, one should continuously deploy the master branch to a staging environment and deploy tagged commits occasionally to the production environment. While this works properly if there are only two environments, I’d also like to make use of the dynamic environments available in GitLab, i.e. for each branch off of master, I want to be able to create a review deployment.

Now if I have e.g. multiple microservices which make up a single application, what’s the best way to deploy all microservices to a separate kubernetes namespace for each branch, when pushing the repository of a single microservice using GitLab CI. I can’t think of a clean solution as soon as there are dynamic environments involved.

How am I supposed to handle the above use case properly for microservices? Are there branching models / git workflows which work better for microservices?

We also switched from Git Flow to GitLab Flow several months ago and so far we're very happy with our decision. It works very well with GitLab, it's easy to understand and less error-prone and you don't need additional Git commands (which are quite slow on Windows).

We use Docker Compose to organize our microservices and have a separate project/repository just for the staging and production compose file(s) of our actual application. In the project and CI of each service you can then combine such compose files with a specific one that just tells Docker to use the specific branch of the specific service (and maybe other branch specific settings).

The production compose file contains the production specific settings and tells each service to use a specific tag:

service1:
    image: examples/service1:1.5.4
    ports:
        - "8000:8000"
    volumes:
        - "/data"
service2:
    image: examples/service2:3.5.2

The staging compose file tells each service to use the image of the master branch:

service1:
    image: examples/service1/master:latest
service2:
    image: examples/service2/master:latest

Then the compose file for a specific branch of service 1 just looks like that:

service1:
    image: examples/service1/some-branch-name:latest

When you deploy a review environment for the specific branch with all three compose files at once you'll get something like the following:

service1:
    image: examples/service1/some-branch-name:latest
    ports:
        - "8000:8000"
    volumes:
        - "/data"
service2:
    image: examples/service2/master:latest