feature flags vs versioning for APIs We are looking at moving to a trunk based development branching strategy rather than gitflow that we currently use. I understand that feature flags are crucial to trunk based development. However, is there still a place for versioning APIs if I am using feature flags to control exposure of new features across UI and APIs?

All of our APIs are are internal to our application and are not exposed directly to  a third party. Even in a case like changes in schema for response/requests - those could be treated as a new feature and handled via a feature flag to enable dark deployments etc. So, what would be a good case to still use API versioning if I have feature flags going?

Will appreciate any thoughts from your experience or any good articles you have come across related to this topic.

Thank you - VK

The two approaches whilst somewhat overlapping solve two different problems:

Feature flags allow you to decouple the action of deployment from the action of release.  i.e. you can push a change to an API that adds functionality to support a new feature, then when your product owner wants to release it all they need to do is toggle the feature on.
API versioning enables you to make changes to your API without breaking compatibility with older clients.

In both cases, you need to ensure that old feature flags and old API versions are eventually removed from the codebase or it will become very complicated very quickly.

The deciding factor comes down to whether or not you have control over all of your clients if you can guarantee that you can make changes to both your clients and the API in such a way that they will remain compatible, or incompatible clients can be removed from your ecosystem very quickly you probably don't need API Versioning.


Code review with GitLab? Is there a way to perform code review (for your own projects) using GitLab? For instance automated reviews, but also manual review where you can comment and propose changes at a review of a commit?

Yes, it's possible. Gitlab has the concept of Discussions (either single comments or discussion threads), which you can have on a variety of development items/activities, including the code itself (in which case they effectively are code reviews):

  The ability to contribute conversationally is offered throughout
  GitLab.

  You can leave a comment in the following places:

  issues
  epics
  merge requests
  snippets
  commits
  commit diffs

But note that only merge request discussions are presently considered resolvable.

Even though the check for a merge request can be configured to be automated, the starting of a review/discussion itself (if/when needed) is a manual operation. From Threaded discussions:

  To start a threaded discussion, click on the Comment button toggle
  dropdown, select Start discussion and click Start discussion
  when you’re ready to post the comment.


Docker mounts files as uid 1001 I have a Docker image that runs a PHP application. Everything works fine on my machine and the ones of my colleagues.
However, some colleagues got a new computer that runs on Ubuntu 20.04 and by default the user on these computers have the user id 1001. If I run the id command on these computers I get 1001 for the user id and group id. So, when they starts the container, the files are chown with 1001:1001 and the code cannot be executed and no files can be written in the document root of the project.
Do you know what is happening ?
Thanks for your answers!

Ideally dont run docker as superuser(root). You should create a user within dockerfile and chown the file system with that user. You can create a user and then add access to that user on the file system. In the below example there is appuser , with ownership to src
FROM ubuntu:latest
RUN useradd -r -u 1001 -g appuser appuser
RUN chown -R appuser.appuser /src
USER appuser
CMD ["appuser", "/src/index.js"]

Its not recommended to run as root
You could also run as any user (Say root id = 0) if needed from externally
docker run -d --user 0 ubuntu:latest

You should never relay on gid/uid  of the linux user outside the container.


Jenkins Pipeline project can't when on branch I want to use when on branch condition. The Jenkinsfile is like this

 stages{
        stage('Checkout project branch master'){
            steps{
                withCredentials([sshUserPrivateKey(credentialsId: 'credential', keyFileVariable: 'CREDENTIAL')]) {}
                git branch: 'develop', credentialsId: 'credential', url: 'git@bitbucket.org:user/demopipelinejenkinsfile.git'
            }
        }
        stage('develop'){
             steps {
                   echo 'on branch develop`
             }
        }
        stage('any_branch'){
             environment {
                   GIT_BRANCH = sh(returnStdout: true, script: 'git rev-parse --abbrev-ref HEAD').trim()
                   }
             steps{
                   echo "${env.GIT_BRANCH}"
                  }
        }
 }

The console output shows that the stage develop is skipped due to condition. However, the last stage still echoes the current branch is "develop"

Question
How to use when with branch in Pipeline project (not Multibranch project)

This is the way to go:

stage('Deploy'){
  when {
    expression {env.GIT_BRANCH == 'origin/master'}
  }
  steps {
    ....
   }
}

Take care, this is only working with the declarative syntax.

The environment step is used to "set up the environment" meaning this is the place to declare environmental variables.

Also, in my case I did not declare the GIT_BRANCH var myself. But jenkins provides it as an environmental (therefore "env.GIT_BRANCH") variable if you did "checkout scm" to checkout your repository. If you want to check all the available env vars, do this as a step: sh 'printenv'

check out this article for more info: https://jenkins.io/blog/2017/01/19/converting-conditional-to-pipeline/


How to get started with CloudFormation? There is an official website and there are multiple ways to learn CloudFormation. Just curious what worked best for you.

Linux Academy has many great courses on CloudFormation as well as other AWS and DevOps tools and services. They also have many guided labs where you will be given hands on guided learning of how to use various technologies.

Otherwise, creating templates and launching CloudFormation stacks is something that you could learn by just creating infrastructure piece by piece in your CloudFormation template (ex. add an S3 bucket to your CFT -> launch stack -> add and EC2 instance -> launch stack -> etc. etc.). The AWS documentation is very helpful along with a good plugin for your IDE.


Command "git [command] --help" opens as HTML in my code editor - no output to terminal. How do I revert to default behavior? When I type git reset --help I don't get output in the bash terminal, but a new tab opens in my code editor to reveal some dense HTML.  How can I change Git's behavior to display the help message in the bash terminal?  I'm not sure how to fix this.
Update:
.gitconfig file:
[core]
    editor = 'C:\\Program Files\\Sublime Text 3\\sublime_text.exe' -w
[user]
    name = name
    email = 'email
    name = 'name

$ type -a git
git is /mingw64/bin/git
git is /mingw64/bin/git
git is /cmd/git

Update 2:
It looks like help.format is set to html, but after my user.name and other info, help.format is set to web (I did this).  I tried setting the browser using multiple browser names, but git fails to launch the help command in the browser and still opens in my code editor.
$ git config -l
core.symlinks=false
core.autocrlf=true
core.fscache=true
color.interactive=true
help.format=html
user.name=name
help.format=web

$ git reset --help
git config option web.browser set to unknown browser: edge
Resetting to default...

This is not the default behavior of git so you have made some local config change to cause this to happen.  Look in your ~/.gitconfig to see if there's anything that doesn't make sense to you.  Try commenting it out.  The core.pager setting is one of the first things I would check.  This stackoverflow question has a variety of options for dealing with this sort of thing.
update 1
I haven't tried to get this sort of thing to work, but according to the docs you can also specify a man.viewer to define the program that is going to run.  Do you get any errors if you try git help -w reset?
One work around would be to get the help to show up in the terminal by using git help -m reset.  By using the help subcommand instead of subcommand --help you can specify whether you want it to try man, info, or web.


Does full automation still provide a better ROI at small scales? Obviously, Automated configuration management and deployment makes sense at large scales when you need to manage hundreds of servers and you need those servers to be identically configured to a consistent standard.

However, in situations where systems are small and need only 2, 3 or 4 servers, it can often be faster in build time to simply stand up a first instance and clone subsequent instances - changing IP addresses and hostnames by hand.

The theoretical advantage to automated deployment and configuration management is that code can be deployed faster, tested faster and services are provided more reliably by automating out the problem of human error. Future upgrades of the underlying OS (eg, CentOS 6 -> CentOS 7) are better/faster and in the long run the above benefits outweigh the initial larger investment of programmatically automating deployments using a configuration management system?

Are there any real-world non-anecdotal studies and data-points to back each of the above claimed benefits to small shops with very low numbers of servers to manage?

There is definite value in automating manual tasks and placing systems under configuration management that is done using code, not using paper and human intervention.

One huge benefit is the reduction in the amount of rework. You can consider any problem caused by a human mistake in repeating the same sequence a case of rework. As well as any bug that needs to be fixed more than once.

Rework is costly, and there has been some research done to quantify its cost. You can read about it in this whitepaper - https://devops-research.com/roi/

The same whitepaper also quantifies the cost of failure for services/systems, and the slowness to resolve failure. All of which are greatly reduced by having a robust automated process that prevents human mistakes from ending up in production systems.

The topic of automating manual tasks has been also investigated in the State of DevOps Report that has been issued every year since 2013. Most of these are available at https://devops-research.com/research.html as well (with the exception of 2013). In every report it was mentioned as the biggest driver for excellence in high-performing organizations of all sizes. There are plenty of reasons and explanations that can be used as proof for your case.


What are best practices for setting SLOs for a new service? I am guiding teams setting SLOs for their services and a question came up about setting SLOs for services that are still in design.
Are there standards for setting SLOs for services that don't exist yet?
Or is the best practice to start the service and base the SLO on initial performance and move towards a goal after?

In an ideal world, the SLOs would be driven by business needs, and information about your customers.

Remember: an SLO should be the level of reliability at which most users are happy with your service.

To do this you can partition your service's behaviour into critical actions.

A critical action is a discrete piece of functionality offered by your service, and which your customers care about. It is typically defined from the point of view of the customer, and independent of the implementation (eg., "Customer successfully added an item to their shopping cart", "Customer viewed the high score table")

With some critical actions chosen, decide what the correct SLI specification should be for each action: does it succeed or fail, and needs an availability SLI; does it take some time to complete, and need a latency SLI; does it produce data to be consumed later, and needs a freshness or a correctness SLI? And so on. (Adding an item to a shopping cart should have an availability and a latency SLI; viewing a high score table should also have freshness SLIs, and potentially correctness)

With these specifications you can start to think about what the impact of unreliability would be on your users: will one failure every few days make a user unhappy? What about one failure per hour? Are failures masked by client-side retries? Will users be annoyed if clicking a button results in a 100ms delay before something happens? What about 1000ms or 10 seconds?

At this point, you have enough information to start setting SLOs and thinking about how to implement your SLI specifications: making trade-offs between fidelity, cost, and complexity. It is here that you can take advantage of the fact that the service is still being designed, and ensure it is designed with observability in mind. For example:

Splitting the service into microservices such that those microservices map neatly on to different SLIs and critical actions
Ensuring that any HTTP requests that should be counted towards different SLIs can be disambiguated at the load balancer (eg., don't have all requests be POST requests to /handler)
Include logs and metrics (eg., via OpenCensus) in the design from the start and consider their impact on your application's scalability and reliability.
If you plan to use synthetic probing, allow some way to do this (consider how much of a web page's logic is in client-side scripts, as opposed to server side, and how you would monitor this)
Can you include client-side instrumentation, and how would you collect the results?

The most important thing when setting an SLO for a new service is to plan to iterate: Even if you don't know what level of service is necessary to make users happy to begin with, you should have some metrics you can use to determine this (satisfaction scores, ticket levels, random surveys) and use these to tighten or relax your current SLO as necessary.

Don't hold your service to an SLO that is harder to defend than your customers require, but do ensure that your SLO is sufficient to keep them happy.


Docker image file location I run Docker on Windows 10 machine with WSL 2 support. I would like to know where docker image files are located in this situation? Where docker image files are located in case when no WSL 2 is enabled.

According to this blog post, Docker Desktop on Windows stores its containers inside another container, that container is stored in C:\ProgramData\DockerDesktop. If you docker inspect a container you have setup, it will give you a Linux path, which you can then get to once you go inside the host container using docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -i sh
This assumes you are using the default (Hyper-V) hypervisor for Docker Desktop. Where you use VMware as the hypervisor instead, different rules may apply.


What are the daily job duties of a devops? I have been working for linux management over 10 years. In the last 5 years, I have added a lot of coding experience through software development like php, ruby, go, etc. I wanna be a devops, but I have no idea how to get into it.

Keep learning other requirements for devops engineering. such as:

Operating System concepts (Process management, threads and concurrency, POSIX basics, I/O management, file systems, memory storage, ...)

as a devops engineer you need to know everything about the operating systems and how they work. to know everything about the operating systems, first you need to be a system administrator or sysadmin.

managing servers

server performance management helps in increasing agility and optimizes the performance of both physical and virtual servers.

service management

IT service management a cultural mindset to ensure that the desired outcome for the business is achieved. It incorporates principles and practices from various management approaches, such as lean manufacturing, organizational change management, system analysis and risk management.

bash scripting (vim, emacs, ...)

every sysadmin needs to have a good knowledge and experience in bash scripting.

working with terminals

every sysadmin needs to live entirely on the command line.

web servers (nginx, apache,...)

every sysadmin needs to know how servers work and how web servers send data.

CI/CD tools (jenkins, circle ci,...)

automation and automated testing improves agility and speed in scale.

infrascructure as code (docker, ansible, kubernetes,...)

IaC can be a key attribute of enabling best practices in devops.

cloud services (aws, azure, google cloud platform, heroku)

Here's the complete 2020 devops engineer roadmap: https://roadmap.sh/devops


What is the best Windows Infrastructure to run Adobe After Effects like AWS Lambda? I have an app where we allow users to create videos by choosing an AE template and modifying the text, images, etc.
We run AE on a windows server which renders the final video.

For other scenarios where we do not use AE, we use AWS Lambda of which we can spin off 100 instances simultaneously and they all use their own resources. Whereas for AE we use a Windows server and all parallel rendering use the same resources and hence very slow.

Are there any PaaS where I can run AE and spin up multiple instances?

In general a platform-as-a-service solution runs the application code on top of a provider-supplied sandbox infrastructure which complements your app code so that together they implement the complete app functionality. But this puts specific requirements (often quite restrictive) in front of the application code, for example:

a limited choice of (typically high-level) programming languages
certain pieces of mandatory functionality in your app code (like health checks, etc)
certain limitations of what the app can do (for example not allowing writes to the local filesystem)

So when looking for a platform-as-a-service solution you need to be looking from both the app perspective and the sandbox perspective.

Now your case brings additional requirements to the game:

running a 3rd party app which you cannot modify, so you need to find a PaaS offering whose requirements are met with the application as-is, unmodified
a Windows-based sandbox - rare, most providers use open-source infrastructure, I suspect primarily because of the licensing terms. Your highest chance of finding such offering would be on Azure, but not necessarily the only one
AE licensing terms - how many AE instances can you run?

Due to the extra requirement on the sandbox a infrastructure-as-a-service solution would be, IMHO, a better fit for your case (you're actually using it presently, maybe just scaling it would be a much simpler solution for you?).

AFAIK the only offering (outside of Azure, with which I'm not familiar) that comes very close to an infrastructure-as-a-service but still is, in a way, a platform-as-a-service is Google App Engine Flexible Environment:

meeting the app requirements might be possible by actually writing an app implementing a rendering service which uses AE under the hood, as a dependency
it supports (docker-based) custom runtimes, you might be able to build one with AE/Windows (maybe how it's done for ASP.NET apps would help)


How do I troubleshoot missing data in my Prometheus database? I have been gradually integrating Prometheus into my monitoring workflows, in order to gather detailed metrics about running infrastructure.

During this, I have noticed that I often run into a peculiar issue: sometimes an exporter that Prometheus is supposed to pull data from becomes unresponsive. Maybe because of a network misconfiguration - it is no longer accessible - or just because the exporter crashed.

Whatever the reason it may be, I find that some of the data I expect to see in Prometheus is missing and there is nothing in the series for a certain time period. Sometimes, one exporter failing (timing out?) also seems to cause others to fail (first timeout pushed entire job above top-level timeout? just speculating).

All I see is a gap in the series, like shown in the above visualization. There is nothing in the log when this happens. Prometheus self-metrics also seem fairly barren. I have just had to resort to manually trying to replicate what Prometheus is doing and seeing where it breaks. This is irksome. There must be a better way! While I do not need realtime alerts, I at least want to be able to see that an exporter failed to deliver data. Even a boolean "hey check your data" flag would be a start.

How do I obtain meaningful information about Prometheus failing to obtain data from exporters? How do I understand why gaps exist without having to perform a manual simulation of Prometheus data gathering? What are the sensible practices in this regard, perhaps even when extended to monitoring data collections in general, beyond Prometheus?

I think your can do some kind of alerting on a metric rate with something like this:

ALERT DropInMetricsFromExporter
  IF rate(<metric_name>[1m]) == 0
  FOR 3m
  ANNOTATIONS {
    summary = "Rate of metrics is 0 {{ $labels.<your_label> }}",
    description = "Rate of metric dropped, actually: {{ $value }}%",
}

The main idea is to alert whenever the metric rate is at 0 for 3 minutes, with the proper metric name and a label somewhere telling from which exporter it comes it should give you the correct information.

Choosing the right metric to monitor by exporter could be complex, without more insight is hard to give a better advice out of vacuum.
This blog post could be an inspiration also for a more generic detection.


Is it a bad idea to use a web app including nginx, gunicorn and flask in a single docker container? I want to deploy a web app as a docker container. Currently I need to set up nginx, gunicorn and flask to realize this. There are many guides around to do it in docker but always divide it in two docker containers:

flask, gunicorn
nginx

Is it possible and a good idea to use all three components in just a single docker container?

While this is possible it is not a best practice.  Whenever possible, containers should contain a single process.  The reasoning for this is:

It fosters a modular design where you can more easily plug and play components.  What if you decide not to use nginx and use asp.net core in the future?  If everything is tightly coupled in one container then the entire container will need to be reworked any time one of these components changes.  Look into microservices for more information on the benefits a modular design.
It is easier to parse container logs as each container will have its own distinct output.
If a process crashes and everything is in one container, the entire container will crash.  If you have them broken out across multiple containers, recovery is simplified.  While this may not be applicable for running standalone containers, it is very useful if the containers are running in a Kubernetes Pod as the Pod can restart.


Managed MySQL cluster - similar solution for a private cloud Is it possible to build a similar solution to managed MySQL cluster on a private cloud (VMware vSphere / Hyper V). Even with Kubernetes.
A solution which supports auto-scaling, auto-upgrade, multi-master and similar to the largest public clouds offerings.

There shouldn't be any reason why you can't.  I think that the issue would be in getting the memory management correct on the host.  The orchestration would be the same.

Here's some resources:

https://dev.mysql.com/doc/mysql-cluster-excerpt/8.0/en/faqs-mysql-cluster.html#qandaitem-B-1-13
https://www.digitalocean.com/community/tutorials/how-to-create-a-multi-node-mysql-cluster-on-ubuntu-18-04.  It shows with docker as well. it shouldn't be hard to apply to vmware or k8s.
https://towardsdatascience.com/high-availability-mysql-cluster-with-load-balancing-using-haproxy-and-heartbeat-40a16e134691/
https://www.cockroachlabs.com/docs/stable/orchestrate-a-local-cluster-with-kubernetes.html to set up k8s locally.


Storing Meta Data for Apps/tools Have created a number of images via packer and soon coming to the conclusion that binaries/images soon rack up in qty and need to find away to store config data.

We are going to use ansible to hold config data for apps and was wondering about how people manage versions of binaries, versions of images, ingress egress data etc etc. I wondered if anyone knew of either a regularly used/industry approach to managing app data, versioning, config data etc.  My default thought was a scheema to use in ansible, or perhaps a tool that can manage this data.

A combo of packer images, binaries and other images I am beginning to think that these files could increase in numbers v v quickly. How do people manage this info??

Problems with golden images are that you have to store a full system image with overhead of data which is not build-specific, and using conventions for configuration to make deploy process the same for all versions.

TL;DR: Infrastructure as a Code, version-specific storage for data outside the app package.

The problem is easily solved by storing config files and templates, data and installation scripts (including ansible playbooks to manage) inside both code and packages/images, so install and initial configuration would always be version-specific (switching to the same commit you are deploying to have configs you need). This could be done with VCS and current package management software (deb, msi, PKG etc. and scripts for unattended deploy and installation) with ease. To avoid using different images of OS and other environment, storing and using a single golden image for all builds is a standard practice: it provides consistency and you can automate CI/CD pipeline for image update process.

If some data is stored outside (i.e. a lot of current Windows apps outside of Store are installed by running a downloader, then installing app) and you have to keep large number of them (no specific branches only like stable-beta-alpha), then storage should be versioned too (i.e. using version-specific hashes as names) and links in install scripts or artifacts would point to the one version they need. But you also have to keep the register of pairs build:data version, so when there's no need to use specific version, deletion of related data could be performed within the same remove process. Also, you have to decide whether you want to store complete images or use some differential systems (incremental or differential snapshots of files/filesystems), latter is much more complex and hard to manage so it's not used widely AFAIK (with an exception for some DBs, i.e. Facebook stores MySQLs snapshots and binary transaction logs so they could deploy or restore exact state up to certain transaction id).


Company is worried about exposing our PII to Tableau, Can we run Tableau without exposing our data to any 3rd parties? I'm looking to understand a bit more from the technical side about how Tableau (The company itself) works with the data that everyone is providing them.

My company is looking to use Tableau as our main visualization tool as we grow larger, but we have a lot PII data, and just confidential data in general that we are not able to expose for compliance reasons to any 3rd parties.

Is there a way to run Tableau, and take advantage of all of the features that it has (Or a majority subset) in a way where we can still make connections to our reporting databases (MySQL/Postgres/RedShift), and also in such a way that a team of Data scientists can all access this information together, but silo it so that none of this information ever touches the Tableau company servers?

I know that most likely in good faith they do not store this information, but that is not enough for compliance reasons for us to work off of.

If I had an inherent mis understanding of the way in which Tableau operates I apologize, would love to learn more!

Thanks for the help

There are several alternatives:

since you are looking to use Tableau as our main visualization tool, you might try other on-premise alternatives, such as:

Pentaho Community Edition
QlikView on-premise

Also, there is on-premise license for Tableau


Looking for good books on Immutable Infrastructure I am looking for the best books on Immutable Infrastructure. I am interested in books on theory and application. Thank you.

Although I am willing to bet that this answer may not age well, I would suggest

"Infrastructure as Code" by Kief Morris for the "theory" part.

For the "application" part, I would suggest "Terraform up and Running" by By Yevgeniy Brikman.


How do I dynamically take variables from previous plays in Ansible? Example, I have 3 tasks, which are registered ec21, ec22 and ec23.

ec22 runs when ec21 fails, and ec23 runs when ec22 fails.

Now, I have another task below it which is :

- name: Add new instance to host group
  add_host:
    hostname: "{{ item.private_ip }}"
    groupname: launched
  with_items: "{{ ec2x.instances }}"

where ec2xis any of the previous tasks.

So, how do I set ec2x here?  cause, Any of my 3 tasks can run. So, it can be ec21 or ec22 or ec23.  So, how do I dynamically write that in the adding task after them?

It feels dirty, but I guess something like that would work:

- name: ec21
  [...]
  register: ec21_result

- set_fact: end_result= "{{ ec21_result }}"
  when: ec21_result|succeeded

- name: ec22
  [...]
  register: ec22_result
  when: ec21_result|failed

- set_fact: end_result= "{{ ec22_result }}"
  when: ec22_result|succeeded

- name: ec23
  [...]
  register: ec23_result
  when: ec22_result|failed

- set_fact: end_result= "{{ ec23_result }}"
  when: ec23_result|succeeded

- name: Add new instance to host group
  add_host:
    hostname: "{{ item.private_ip }}"
    groupname: launched
  with_items: "{{ end_result.instances }}"

Basically setting the end_result as soon as you succeed a play.


How I can retrieve aws lambda cloudwatch logs in txt format so I can search on them? A lambda that I have deployed has gone wrong. Therefore, I want to look on logs and find out whats wrong about it. Hence I look upon cloudwatch logs for any error message.
But the cloudwatch environment allows no easy way to search for a string in logs. Therefore I need to download a portion of it in text format so I can grep string in it. Do you know how I can do that?

There are several ways for exporting logs from CloudWatch in searchable text format:
The easiest way of downloading logs in text format is to go to a log group -> Select a log stream -> go to Actions and select Download search results as CSV:

This will download the logs in a .csv format which may or may not be what we want, nevertheless I think it is good for searching a analysis.
A more involved way to export logs would be to select a log group and from the Actions and for that log group select Export Data to S3:

This may export every stream from the log group to an S3 bucket, from which we can download the logs in text format. In order to be able to write to an S3 bucket, we need a policy. There is a full documentation on the AWS docs about how to do this.
I need to mention the CloudWatch provides a way to analyze logs without being able to export them by using CloudWatch Logs Insights. CloudWatch Logs Insights has its own query language.


Kubernetes flannel pod getting the wrong network Flannel running in a pod is getting the wrong subnet and networking is just not happy, the symptom is flannel is being assigned /24's from the 10.105.0.0/16. it should be assigning /26's from 10.105.5.128/21. Thanks for any help.

here are the details:

/usr/bin/kubeadm init \
--kubernetes-version v1.7.5 \
--pod-network-cidr 10.105.5.128/21 \
--service-cidr 10.105.5.136/21 \
--token XXXXXXXXXXX

kube-flannel-rbac.yml is loaded after kube-flannel.yml
only modified bit(SubenetLen and Network) from kube-flannel.yml:

{
  "Network": "10.105.5.128/21",
  "SubnetLen": 26,
  "Backend": {
    "Type": "vxlan"
  }
}

DNS is set in the systemd file to:

--cluster-dns=10.105.5.136.10

Using Ubuntu 16.04 LTS and stock kernel

here is the docker daemon.json file:

{
"hosts":[
    "fd://",
    "0.0.0.0"
],
"ip-masq":false,
    "experimental": true,
"registry-mirrors": [
    "http://hub.xyz.com"
],
"insecure-registries": [
    "http://hub.xyz.com"
],
"tls": true,
"tlsverify": true,
"tlscacert":"/etc/docker/ca.pem",
"tlscert":"/etc/docker/cert.pem",
"tlskey":"/etc/docker/key.pem"
}

all kuberentes components are 1.7.5 installed from ubuntu k8s repos

here is the log of the kube-flannel container:

I0926 03:29:10.214198      89 main.go:446] Determining IP address of default interface
I0926 03:29:10.216166      89 main.go:459] Using interface with name eth0 and address 10.105.5.12
I0926 03:29:10.216261      89 main.go:476] Defaulting external address to interface address (10.105.5.12)
I0926 03:29:10.242216      89 kube.go:283] Starting kube subnet manager
I0926 03:29:10.242055      89 kube.go:130] Waiting 10m0s for node controller to sync
I0926 03:29:11.242864      89 kube.go:137] Node controller sync successful
I0926 03:29:11.242957      89 main.go:226] Created subnet manager: Kubernetes Subnet Manager - kube-m2.XXXXX.com
I0926 03:29:11.242969      89 main.go:229] Installing signal handlers
I0926 03:29:11.243984      89 main.go:330] Found network config - Backend type: vxlan
I0926 03:29:11.288902      89 ipmasq.go:51] Adding iptables rule: -s 10.105.0.0/21 -d 10.105.0.0/21 -j RETURN
I0926 03:29:11.306692      89 ipmasq.go:51] Adding iptables rule: -s 10.105.0.0/21 ! -d 224.0.0.0/4 -j MASQUERADE
I0926 03:29:11.314413      89 ipmasq.go:51] Adding iptables rule: ! -s 10.105.0.0/21 -d 10.105.0.0/24 -j RETURN
I0926 03:29:11.323229      89 ipmasq.go:51] Adding iptables rule: ! -s 10.105.0.0/21 -d 10.105.0.0/21 -j MASQUERADE
I0926 03:29:11.329676      89 main.go:279] Wrote subnet file to /run/flannel/subnet.env
I0926 03:29:11.329746      89 main.go:284] Finished starting backend.
I0926 03:29:11.329829      89 vxlan_network.go:56] Watching for L3 misses
I0926 03:29:11.329903      89 vxlan_network.go:64] Watching for new subnet leases

You have either a bad understanding or misconception about how a cidr works.

10.105.5.128/21 is just one address in the range 10.105.0.0 - 10.105.7.255.
So either your cidr bits are not the one you're expecting, 10.105.5.128/26 would be a valid network definition but that's a /26 subnet within the 10.105.0.0/21, or you're not using the proper network definition.

You can check those division on this very helping site (the link already use your networks above.


Proper way to showcase multi-container web app to customers I have a web app consisting of multiple services (authentication service, database, main web app..), and each of these services can run in container.
Currently, when a customer ask for a demo, I am creating and running new instance of these containers on new open ports.
But, it seems like this is not the right way to do, because each container is just consuming 1 GB RAM of my host machine (currently total 32 GB), even it is not used by anyone.
What is your suggestions and is there any on hand solutions for this scenario?
What should I research to find a solution for my need?

Edit response to comments:

Firstly, it is not a laptop :), It is just a VM on Azure(32GB and 4 CPU),
actually, yes,  it's like laptop :(
My customers want to (actually I want) test/evaluate my app without any installation, just filling a demo request form and then I will send them a mail to say "your main web app is running on http://somehost:someport, you can use it now!"
Currently, I can perform the situation mentioned above, and someone who wants to use my app can use it. But it is not efficient.(???)
You mention swarm/kubernates, So my main question is that; Is solution for my need is should be using swarm? If yes, How? is there an best practice approach you can share with me?

Now I will search on how to make my own SaaS system:)

You mention that you are demonstrating this for a customer. The first question you should ask yourself (or, more pertinently, the customer) is :

  what does this customer really want?

Why have they come to you for this work? What is the pain they are currently feeling, what value are you providing to them.

Then, compare and contrast - show them the delta that your solution brings to them.

It may be for example that they want to maintain a legacy application, but upgrade the hardware or underlying components ? In that case, show them how you have contained them with no negative experience to their users.
It may be that they need to scale the capacity for usage of their application   - they want to be able to meet demand elastically. Show them how by deploying in a container setup, you can scale out.
It may be that they need four nines reliability - show them that you can have an easier HA setup - knock out one of the instances and prove that the application still works.
It may be that they want faster delivery with no little impact on quality. Show that your delivery pipeline works smoothly with some automated tests and QA, with delivery and deployment.

TL;DR : Find out what your customer wants. Demonstrate the existing way of doing things. Compare it with the "new" way, and highlight the delta.


Local Prometheus and Grafana Installations to Monitor a Kubernetes Cluster We want to start our monitoring off the cluster on a local server.  Is this practical and will it be feasible and reasonable to hook local Prometheus and Grafana installations into a Kubernetes cluster?

It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen kube-prometheus? It's a one-shot install that will get you 90% of where you need to be for your cluster.

The only way I can advise it would be if you're going to run 5 or more clusters and want central monitoring, but I'm a bigger fan of having a Grafana per cluster.

Edit: Getting started with kube-prometheus is a one command install. kubectl apply -f manifests/


How to write logs for certain tasks only? I am using Ansible and created a Playbook for deployment. I defined log_path in ansible.cfg and now tasks output are going to this log file. I want to send only few task's log into this log file. How can I achieve this ?

One option is to use no_log: True attribute but this is not displaying the logs on console.

So my requirement is to display all task's logs in console but I want to send only few task's log to my log file.

It's in the wish list, I think. Quoting from the doc log_plays - write playbook output to log file

* This callback writes playbook output to a file per host in the /var/log/ansible/hosts directory
* TODO: make this configurable

IMHO required functionality is not available at the moment. Take a look at the available plugins. "You can only have one plugin".

ansible-doc -t callback -l

There is an option to customize a callback. context_demo looks promising.


How do I setup ssh from jenkinsci/blueocean docker to remote? I am running jenkinsci/blueocean docker locally as per the instructions of running jenkins using a docker on jenkins.io. Everything seems to be working correctly but when I docker exec into the docker, I can't use sudo and I cant create a home directory for the jenkins user with an .ssh directory.

There seems to be apk package manager but if this functionality is not out of the box it makes me wonder if I am doing this wrong.

Does anyone have any experience setting up a basic ssh config on this docker image?

You can run an ssh server in the container as the ENTRYPOINT. It can either be started in the foreground :

ENTRYPOINT ["/usr/sbin/sshd", "-D","-p","22"]

Ensure that you have openssh installed and host keys setup.

Perhaps this Ansible role would be illustrative


How to trim spaces for parameter value in jenkins pipeline? I am using jenkins pipeline job with parameters. when i had any trailing spaces in parameter value the job gets failed. is there any way to trim the spaces in pipeline job to avoid build failure?

Use the Groovy trim() method.  For instance, def myTrimmedParam = params.myParam.trim().


Methodology for software development and deploy in fast-paced agency Background:

Three months ago I was called to do a freelance job for a rather big agency, with 40~50 employees.

Things were chaotic when I got here. The entire development team had been hired by another company one month ago, and there were a lot of overdue jobs. I freelanced here for about three weeks, until they hired me. Since then, I had been working hard to fix things, and there were so many things to do that I had to let go of the process to focus on delivering the software.

Now, three months later, the dust have finally settled down, and now I'm looking forward to implement good development practices and methodologies, mixing development, deployment and project management.

Scenario:

We have several different clients, most of them with their own servers. Some of them have their own infrastructure team to handle their servers. Projects are in different languages, and I have little control over the client's servers.

Throughout the day, there might pop up several demands for different clients, and I must deliver them as quick as possible.

Conclusion:

I have to deliver things quickly, but I feel a urge to employ a methodology to do it, because I know that if I just keep doing things without proper control, some thing will certainly go wrong sooner than later.

My two cents:

One of the first things I plan to do, is to elect a preferred development language and framework, so we can specialize in something. Other than that, I'm not sure what to do.

In terms of development procedure, you should consider using the agile method. This is the most widely used method of software development and has proven itself. Within the scope of the agile method Pair programming will allow you to spool up another developer to assist with your projects when you are ready and find that you have more work than you can handle. You can then move into a standard agile method like scrum/kanban/scrumban (or continue the pair programming if you find you like it!)

One particular reason for choosing this method is that is lends itself well to DevOps and Continuous Integration. In fact, one could describe DevOps as a specific way of implementing the agile method. The Iterative, incremental and evolutionary philosophy of agile and the very short feedback loop and adaptation cycle marry beautifully with DevOps/Continuous Integration and provide a solid foundation to build upon.

  Here in the agency we have a infrastructure team as well, with the possibility to set-up our own internal or external servers to handle things.

Make good friends with the infrastructure team. Involve them early and often and drive them towards a Infrastructure/Configuration as Code philosophy which will allow you to collaborate closely with them. While it might seem like it is faster to just do it manually, doing so is creating technical debt. In a few years when it comes time to upgrade, no one will remember how the thing is built. It will also allow the infrastructure team to look good. Never again will they make a mistake or forget a step during deployment. Either the deployment code works or it doesn't and it will deploy exactly the same way it deployed in test/staging - making deployments and upgrades easy and horizontal scaling a breeze. Ignoring these techniques also slows down your development. If you can just fire up a new server, have it pull down (and compile) your latest git commit, run through an automated testing suite and then E-mail you the results while you go do something else it is a huge time-saver.

If you can, utilize the cloud - with Amazon for instance, a lot of the tools are already built for you. While this might make your Infrastructure team leery, just remind them that the cloud is just someone elses computer. All the cloud does is provide you with hardware. It will still require someone from the infrastructure team to install maintain and operate it. And it will allow you to avoid scalability issues. The database problems, storage and backup issues and headaches have now been outsourced.

  and I have little control over the client's servers.

See what you can do about this. Consider leveraging a tool like The Forman's Katello scenario (I like to deploy it with Salt Stack personally - I don't like ruby) which will allow you to manage and deploy your VMs on nearly any iron or cloud (including bare metal), manage the configuration of that hosts using Salt Stack, Chef, Puppet or Ansible, manage your docker containers and manage updates to your machines. This will allow you to then easily provide the code to your client to either A) automate or B) manually deploy their updates and deployments. Once you have proven the value of the speed and efficiency this method gives you, you may be able to simply use these same tools to spawn your client a copy of this stack for managing their environment or you may be able to talk/upsell your clients into letting you manage these deployments - The Foreman will allow you to configure sites and organizations (as well as mange users if paired with FreeIPA) so you can configure multi-tenancy. And the project is backed and supported by Red Hat.

If you have any doubts about the above, I recommend you pick up a copy of "The Phoenix Project" of the Audiobook which illustrates very well the value and benefit of the above.

  One of the first things I plan to do, is to elect a preferred development language and framework

If you go this route, just make sure to standardize on a language that plays well with REST most DevOps stuff uses this interface.

And finally,

  The entire dev team had been hired by another company - and they didn't have a solid process as well, only a few gits here and there

Do what you can to merge these together into one place. It will make your life a heck of a lot easier.


Which ways are there to decouple deployment from release? One method of continuous deployment is to decouple deployment from release, i.e. deploy updates without activating the changes immediately.

I know that feature-toggles can be used for this, but I'm wondering if there are other techniques for "non-features".

For example, would you build a feature toggle for a bugfix? Probably not and one could argue that bugfixes should be deployed as soon as possible, because it can only get better. And after a bugfix has been released, I sure don't want to toggle it off anymore.
But is this the case? It might be a risky change that you want to release in a controlled way. And if there are unexpected side effects, it's good to be able to roll it back. So, feature flags for every change?

And what about visual changes? For example, can you implement something like a feature flag in CSS? Does it even make sense?

For software in the web app category, depending on you infra/hosting provider such decoupling might be possible to switch incoming traffic across (or split it between) different deployed versions of the sw, practically covering any of the changes you mention: bugfixes, visuals, etc.

Such support would typically not require feature toggles. And it might be applicable regardless of the app being monolithic or split in microservices.

For example Google's App Engine Paas offering has support for traffic splitting and migration.

From Splitting Traffic:

  You can use traffic splitting to specify a percentage distribution of
  traffic across two or more of the versions within a service. Splitting
  traffic allows you to conduct A/B testing between your versions
  and provides control over the pace when rolling out features.

From Migrating Traffic:

  Traffic migration switches the request routing between the versions
  within a service of your application, moving traffic from one or more
  versions to a single new version.


Can Jenkins master connect to a slave node in a private subnet? We've got a Linux Jenkins master node  set up in AWS behind a load balancer and I'm attempting to connect to Windows slave nodes in private subnets.  Is this possible?  The Jenkins URL is accessible when I RDP into each server, but when I attempt to configure the node using "Launch agent from browser" an error occurs.

hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <JenkinsURL>:39057 (retrying:4)
java.io.IOException: Failed to connect to gpa-jenkins-
prod.aws.baxter.com:39057
    at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242)
    at hudson.remoting.Engine.connect(Engine.java:686)
    at hudson.remoting.Engine.innerRun(Engine.java:547)
    at hudson.remoting.Engine.run(Engine.java:469)
Caused by: java.net.ConnectException: Connection refused: connect
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Unknown Source)
    at sun.nio.ch.Net.connect(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
    at java.nio.channels.SocketChannel.open(Unknown Source)
    at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203)
    ... 3 more

What could be the source for this refusal?

As described in Jenkins documentation https://jenkins.io/doc/book/managing/security/#jnlp-tcp-port, you need to configure:

Jenkins to use fixed TCP port for JNLP agent, for instance 39057
AWS loadbalancer to listen on port 39057 and forward it to Jenkins on same port


AWS Credentials - What are multi region user/role management best practices? In the past, I've usually had separate accounts for each region that I've worked in.  These accounts had different IAM roles/etc, so there was very little chance I'd inadvertently mess up region A when working on region B and vice versa.

Recently, we deployed to another region within the same account.  This means, without extra controls, just setting the default region differently on your CLI could have fairly disastrous consequences.

While IAM is global, is it generally recommended to have distinct uses per region / restrict which resources they can hit with policies, or is that overkill?  Are there any good tools for helping you "not mess up" when running anything on the CLI?  (including things like terraform)?  Do you know any other "best practices" or good resources to learn about this particular area?

Start by reading this: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html

Those are best-practices and do not take into account business policies.

I don't think having one account per region is more secure than having policies scoped by regions. In cli, you will have to specify the region in the command anyway (not sure if AWS cli uses a default region if you don't specify one in your credentials file).


Kubernetes API Server KUBE_SERVICE_ADDRESSES parameter and Pod Network configuration Question

Where is the Kubernetes documentation regarding KUBE_SERVICE_ADDRESSES parameter of API server and Pod network configuration?

Background

KUBE_SERVICE_ADDRESSES parameter value needs to go hand-in-hand with CIDR configuration of the Pod network, because KUBE_SERVICE_ADDRESSES is
"Address range to use for services" and I suppose services communicate in the Pod network.

Trying to find out a Kubernetes document which explains it but looks no specific documentation. Please suggest if this is true and where is the documentation related.

/etc/kubernetes/apiserver

# The address on the local server to listen to.
#KUBE_API_ADDRESS="--insecure-bind-address=127.0.0.1"
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

# The port on the local server to listen on.
# KUBE_API_PORT="--port=8080"

# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

Related sites

Using kubeadm to Create a Cluster (3/4) Installing a pod network says specific CIDR needs to be specified to kubeadm init as a option --pod-network-cidr.

Deploy a Multi-node Kubernetes Cluster on CentOS 7:

  Create a flannel-config.json file that will define the Flannel settings. The subnet specified in the Flannel settings should match that of the API server KUBE_SERVICE_ADDRESSES value. This is the subnet that the containers will use.

https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"


Jenkins Plugin:Extended Choice Parameter from JSON file Extended Choice Parameter plugin query: How to load params dynamically from a JSON file for jenkins build job. Here is my jenkins script to add normal parameters and Extended Choice parameters as well.

p

roperties([
        [$class: 'BuildDiscarderProperty', strategy: [$class: 'LogRotator', numToKeepStr: '500']],
        parameters([extendedChoice(
        defaultValue: 'One,Two,Three,Four',
        description: '',
        multiSelectDelimiter: ',',
        name: 'SAMPLE_EXTENDED_CHOICE',
        quoteValue: false,
        saveJSONParameterToFile: false,
        type: 'PT_CHECKBOX',
        value:'One,Two,Three,Four,Five,Six,Seven,Eight,Nine,Ten',
        visibleItemCount: 10)
        ])
])

This is working. But I want to be able to load parameters from a JSON file (using Extended Choice Parameter plugin). How can I modify the above script to accomplish that? PS: Here is the reference for Extended Parameter plugin. https://plugins.jenkins.io/extended-choice-parameter/

EDIT: Updating my answer to include steps on using the Declarative Directive Generator.

You can experiment with the different available configuration options, and sometimes gets hints as to what certain settings provide. For this specific case:

Navigate to <$JENKINS_URL>/directive-generator/
Select parameters: Parameters from the Sample Directive drop-down.
Click the Add button and select extendedChoice: Extended Choice Parameter
Select JSON Parameter Type and then experiment with the available options.
When finished, click Generate Declarative Directive to generate the code block to paste into your pipeline.

You can add the javascriptFile parameter and change the type to PT_JSON.

Using your example:

properties([
    [$class: 'BuildDiscarderProperty', strategy: [$class: 'LogRotator', numToKeepStr: '500']],
    parameters([extendedChoice(
    defaultValue: 'One,Two,Three,Four',
    description: '',
    javascriptFile: '/path/to/file.json',
    multiSelectDelimiter: ',',
    name: 'SAMPLE_EXTENDED_CHOICE',
    quoteValue: false,
    saveJSONParameterToFile: false,
    type: 'PT_JSON',
    value:'One,Two,Three,Four,Five,Six,Seven,Eight,Nine,Ten',
    visibleItemCount: 10)


Octopus Deployment - only deploy some specific folders We are using Octopus for our deployment to different environments. But we found that it always packages the complete solution and deploys it to the next level.
Is it possible to put some restrictions - like do not copy and deploy the config folders.
In our case we just need to deploy some specific folders and not the complete packaged solution. Is this possible with Octopus.

Yes.. we can control the packages using nuspec file which will be added to your solution and you will be specifying the list of folders/files you dont wanna deploy. And you will use that nuspec to create the nuget package for deployments


Building .Net apps outside Docker? My company uses Azure DevOps and utilized Pipelines for building, testing and deploying (currently NuGet packages) .Net Framework apps.

My boss has made it clear he wants us to containerize all our microservices and at some point start using Kubernetes. During this process we will also port our apps to .Net Core, one by one.

After I had made a multistage Dockerfile (building and runtime, but not testing yet) for one of our microservices, he suggested we could simplify it by continuing to use Pipelines to build and test the apps and in a later task add the compiled app to a Docker container in Azure DevOps. This seems to be an anti-pattern as far as I know (I may be wrong), and I understand his desire for simplicity.

Is this an anti-pattern or is it a good idea?

I'd say it is an anti-pattern as you lose the portability of Docker.  Now you will be relying on the build server's configuration instead of portable base image.

If reducing complexity of the Dockerfile is the goal why not do something like this:

Build the .net core project in Docker (eliminates reliance on the build server).
Run the tests (inside or outside of Docker).
Now push the image and run it.

Basically break your Dockerfile into three discrete pipeline steps.  This allows the build to fail on a discrete step.  It also reduces the complexity of the Dockerfile (at the cost of increased complexity in your pipeline).


Storing my Lambda Functions on GitHub Will it be practically possible to build some pipeline;

starting from rapid AWS Lambda function development using blueprints,
to store them later on a new GitHub repo,
that is linked to some CICD such as Travis,
that will in turn deploy successful builds back in AWS Lambda?

Your idea will work. The organization I work for maintains some open source examples how to do deploy to AWS using a pipeline. In comparison we use GitLab for the purpose. Instead of Travis we use GitLab Runner. You configure it using the file .gitlab-ci.yml which runs the tests and deploy the AWS Lambda to dev, test and production environment every time a commit is done.  You store your secret account variables in the CI/CD settings of the repository.

There are the usual drawbacks of AWS Lambda such as difficult or impossible to test and run locally and waiting for deployments takes time, but your deployment process will be good practice.


Jenkins Not pulling the latest commit into master or slave I have a Jenkins pipeline where after every build I change a file(increase build version in a property file) and checkin back to the git repository. After a build, I commit some differnt code change to the git repo and run the build. One of the following two things happen

Jenkins does not pull the latest code change from git to master machine or slave machine.
The build increment checkin stage fails when we checkin from Jenkins slave.

The only solution is to wipe up the repository and build it again . Our project is a very big one and doing this every time causes a lot of time-wasting.
What configuration am i missing?

You can use a plugin called "Workspace Cleanup Plugin" which wipes out the workspace post build. This can be a part of the pipeline and would not require manual cleanup every time when a build should be triggered.

For the second issue, I would recommend you to check if you're able to checkin to the same repository with git credentials configured in jenkins settings through the command line in the slave.


Jenkins won't substitute variables when trying to check out from perforce I want to generalize my code so that it can pull from perforce for any files marked with a type "perforce" because some files need to downloaded from the artifactory instead. I want to sub in the variable perf[0].ID into the 'view' parameter so that I can pull different files depending on the ID. However, when I run it, it's passed in as an empty string. Does the checkout plugin not have any support for variables? If there is no support, would having a spec file give me this generality I want? Or do I need to bite the bullet, abandon the checkout plugin and make a batch script?

-->I have an array perf with a type,path,ID, and size field. Here perf[0].ID = "Linak".

stage ('Perforce binaries') {

        steps {
        //can do this "for perforce[].size()" on specified path
          script {
            for (int i = 0; i < perf.size(); i++ ) {
              dir("raw/${perf[i].ID}") {
                checkout perforce(credential: 'perforce_credentials', populate: autoClean(delete: true, modtime: false, parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'], pin: '', quiet: true, replace: true, tidy: false), workspace: manualSpec(charset: 'none', cleanup: false, name: 'jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}', pinHost: false, spec: clientSpec(allwrite: false, backup: false, changeView: '', clobber: false, compress: false, line: 'LOCAL', locked: false, modtime: false, rmdir: false, serverID: '', streamName: '', type: 'READONLY', view: '"//depot/Software/${perf[i].ID}/..." //jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}/...')))
              }
            }
          }
       }
    }

Your string isn't being interpolated at the correct time because it's in single quotes.  Instead it's being passed to the checkout method as an un-interpolated string.  By the time the checkout method interpolates the string, the perf variable is out of scope and so interpretation of this variable falls back to a blank string.  Something like this is what you (probably) want instead:

view: "\"//depot/Software/${perf[i].ID}/...\"" + '//jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}/...'

Note how this concatenates two strings, a double-quoted string and a single-quoted string.  This ensures that each string gets interpolated at different times.


Ansible Jinja template if statement This is a snippet from my Ansible jinja template which populates an environment specific template.

docker_compose_mq: <string-passed from Jenkins>
docker_compose_profiles: "string"

{% if "{{ risk_docker_compose_mq }}" == "string" %}
  {% "{{ risk_docker_compose_profiles: "string1" }}" %}
{% endif %}

This fails with a pretty generic error message:

"Syntax Error while loading YAML.\n\n\nThe error appears to have been in 'True': line 26, column 2, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\n(could not open file to display line)"}

I'm almost certain it's to do with escaping the quotes here but can't for the life of me work out what I'm doing wrong, any ideas?

The syntax failures are caused by the presence of the {{...}} expression blocks (normally used for filling the template output with the corresponding content) inside the {%...%} statement blocks.

I only used standalone jinja2 templates, so I'm not 100% certain if this applies to Ansible jinja templates as well, but I suspect so. In the Jinja2 {%...%} statement blocks variables are referenced directly (and variable assignments are done in {% set ...%} statements), so what you're after may be along these lines:

{% set docker_compose_mq = <string-passed from Jenkins> %}
{% set docker_compose_profiles = "string" %}

{% if risk_docker_compose_mq == "string" %}
  {% set risk_docker_compose_profiles = "string1" %}
{% endif %}


How to use Vagrant and Docker-for-Windows on a Hyper-V system? Back in the day, I was using VirtualBox (VB) and Vagrant (V) to run VMs. Since I installed docker-for-windows (DfW) and run V, VB complains about Hyper-V (HV).

[user@localhost ~]$ vagrant up
Stderr: VBoxManage.exe: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component
ConsoleWrap, interface IConsole

Although this link indicates that V supports HV, the above error message is contradictive. Why does VB look incompatible with HV, why V says that it supports HV and how to prevent that HV needs to be disabled in order to use V again.

| Product    | Version |
|:-----------|--------:|
| Vagrant    |   2.0.1 |
| VirtualBox |   5.2.4 |
| Windows    |      10 |

Discussion

The suggestion to switch Hyper-V on and off is not the preferred solution as it is time consuming.

If it would be possible to run V and DfW on the same system by replacing VB this could be an option.

Confirmed by this answer on SO referencing  Microsoft technet once Hyper-V is activated it mask the VT-X instructions of the processor to the OS on top of it and windows run like a guest.

Docker for Windows need hyper-V to work, so disabling it is not really an option.

There's no way to have virtual box running on top of Hyper-V and that's quite normal, you should not run an hypervisor within an hypervisor, they'll fight to orchestrate hardware acces.

The best workaround in my opinion is to create hyper-V virtual machine from vagrant with the hyper-V provider instead of virtual box machines, this way you can create VM or container on the same session.

Side note, vagrant already have a note about this in the hyper-v provider documentation page (found while searching the link to add above):

  Warning: Enabling Hyper-V will cause VirtualBox, VMware, and any other virtualization technology to no longer work. See this blog post for an easy way to create a boot entry to boot Windows without Hyper-V enabled, if there will be times you will need other hypervisors.


How to connect a Bamboo remote agent from within Intranet We have a Bamboo server running in the cloud and we would like to connect the machine that is in our local network. The problem is that this PC only has access to proxy that has only port 80 available and company would not allow to open any ports.

We want to avoid setting up new Bamboo instance in our local network.

So I am assuming that

1) you have a PC on your LAN.  The PC can only talk to port 80 outgoing via a proxy

2) you have a bamboo server running a cloud service

3) you wish to initiate a connection from the bamboo server to the PC

Don't get your hopes up but this answer https://stackoverflow.com/questions/19161960/connect-with-ssh-through-a-proxy

explains how to use a http proxy to tunnel ssh.  Unfortunately most http proxies disallow the CONNECT method

Once you have the ssh to the remote cloud service set up then try it again
with a -R option to allow a reverse tunnel

Then you should be able to connect via the reverse tunnel back to the Lan PC


Jenkins cannot reach nodes after upgrade I just upgraded Jenkins from 2.39 to 2.57. After the upgrade, Jenkins can no longer communicate with my SSH build nodes. I get the following error:

<===[JENKINS REMOTING CAPACITY]===>channel started
Slave JVM has not reported exit code. Is it still running?
[04/28/17 01:29:32] Launch failed - cleaning up connection
[04/28/17 01:29:32] [SSH] Connection closed.
ERROR: Connection terminated
java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335)
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804)
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)

Based on Googling 'Slave JVM has not reported exit code.' it seems like a Java version mismatch (JENKINS-30561, JENKINS-29120), but I can confirm that my build nodes are using Java 1.7:

$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

Does anyone have ideas what could be causing this? More info can be provided on request.

I found out via this StackOverflow question which pointed me to JENKINS-27624 which indicates that Java 8 is now required.

Update: I upgraded Java on all my nodes this morning and they are up and working.


How to check minimum code coverage in pull request changes? I want to force the developers to write unit tests for all new code they wrote. The code is written in python. I want to do it in the Jenkins pipeline. And here is a question. Is there any available Jenkins plugin that can do that? Or python library? I want to avoid sonar qube because I think, it's a little bit complex solution for the problem. I was thinking about a solution, that will detect changed files and check their code coverage percentage in generating a report.

I found a solution to my problem. There is a library diff-cover which does exactly what i need. Here is a link -> https://diff-cover.readthedocs.io/en/latest/README.html


How to implement the Load Balancing in Openshift ? I have deployed my application on cloud using Openshift. I tried few blogs but could not find out the best ways to implement the Load Balancing . Please help.

The simplest way is to find the icon that scales the application up and click it. In the current UI it's in Project --> Overview --> expand the application info (">" icon to the left), --> click "^" next to the blue circle to the right.

This gives you a default load-balancing across as many PODs (instances) of the application as your resources and imagination allow.

From the command line, this would scale it to 3 PODs:

oc scale --replicas=3 rc yourapplicationname

Another way is to deploy the same application with a different name, and then edit the route (Applications --> Routes --> RouteName) that points to it. There should be an option:

  Alternate Services / Split traffic across multiple services

This gives you control over how much traffic you want through each version. Good for A/B testing or canary releases.


GitLab - trigger pipeline to build on specific runner Scenario

Create a new virtual machine with runner installed
Trigger a pipeline to run (from another project), which runs on this new virtual machine
Do something else, or fail if pipeline above fails on the new virtual machine

Question

I'm using GitLabs Triggers API to run a pipeline from a project, however would like it to run on this newly created virtual machine.

What would be the best way of parsing this information to the GitLab API, or would something like tags be the best way of doing this?

.gitlab-ci.yml

stages:
  - deploy to test
  - test
  - deploy to prod

Terraform Deploy To Test:
  stage: deploy to test
  script:
    - cd test_deploy
    - python deploy.py

Testing:
  stage: test
  script:
    - curl --request POST --form token=TOKEN --form ref=master http://gitlab.example.com/api/v4/projects/12/trigger/pipeline

Terraform Deploy to Prod:
  stage: deploy to prod
  script:
    - do something if the above testing succeeds, or fail

The above gitlab-ci file would generate a new build machine and deploy it to the "test group" within vsphere with a python script. (Deploying it would activate a gitlab-runner and tie it to a specific project and have some tags associated with it).

To test, it then triggers a pipeline call using the GitLab API, and I would like that pipeline to run on the new generated build machine on vsphere.

Would a work around maybe be something like using gitlab runner exec command or using environments?

There currently isn't a solution for building on a specific runner in GitLab, but there is an issue open for Sticky Runners, which hopefully will be out in the next 3-6 months according to the Milestones!

The work around I've done so far to build a project on a specific runner is to use the GitLab Runner API, in a rather hacky way, along the lines of:

Get all project runners
As I know I've deployed the latest runner, that would have the highest runner "number"
Pause all the other runners associated with the project in question
Trigger the pipeline to build on the latest runner
Poll the GitLab API to get the status of the pipeline
Once that succeeds, resume all other runners!
If the pipeline fails, remember to resume the paused runners...


amazon ec2 ip whitelisted on a third party service provider Currently we are running an application in a single amazon ec2 machine. On our application we are using some third party solution. For using those solution, we have asked the third party service provider to whitelist our ec2 machine ip.

our EC2 machine: IP - 172.12.32.55
Third party solution like global.bank.com and we have told global.bank.com to whitelist our IP

Now we want to scale up our application. We are planing to setup multiple node (our application in multiple ec2 machine) and a load balancer.

So my questions is, how i can manage our third party solution provider? Should I give all our ec2 machine ips to our third party solution providers for whitelist? Or is their any other solution to handle the situation?

A possible approach could be:

Put your instances in a private subnet
Route incoming traffic through a load balancer
Route outgoing traffic through a NAT

Outgoing traffic will then use the elastic IP assigned to your NAT as its source IP.
An added benefit of this approach from a security perspective is that your instances won't be directly reachable from the internet, thus reducing your attack surface.


Cloud agnostic infrastructure automation better than terraform? Are there any infrastructure automation technologies which do a better job at being portable than terraform?

We heavily leverage terraform as it is a cloud-agnostic technology (vs, say, ARM templates in azure, or cloud formation in AWS).

I was very depressed a year ago when we deployed an app from azure to AWS to see that a basic scale-set + load balancer basically required a complete rewrite between Azure and AWS though.  Having used ansible equivalents, the same seems to be true there. Both basically use cloud-specific extensions to create cloud resources meaning that, while you understand the tool still, any project you code has to be re-coded for deployment in a new cloud.

I understand that a generic tool would  have to understand numerous differences like:

Azure uses resource groups to encapsulate all resources.
AWS has security groups to define network boundaries.
AWS load balancers require separate target groups/listeners vs Azure's  simpler backend pool concept.
IAM differences, and so on.

I'm surprised there aren't any companies or OSS projects trying to simplify/automate the translations here with a higher level DSL or something though.  So, I'm thinking I might be missing something.

Note: We are heavy kubernetes users, so I do understand that kubernetes mitigates most of these issues once you've brought up a cluster (and if you have ported all your apps to it).

There are no "agnostic" products due to the dramatic differences of the design of the Cloud Services provided by different companies.

My best recommendation is to write your code to support the different cloud providers. Whether you choose Ansible or Terraform this can be done with Ansible Roles or Terraform Modules that are specific to the cloud provider. Just the same as you would write Ansible roles specific to Operating Systems or different scenarios.


Utilizing telnet with Ansible Was wondering how to best utilize telnet, rather than SSH, with ansible.
We are using a Lantronix term server, and must telnet into individual ports.

More specifically, would it be best to use a specific telnet module? If so, which is most intuitive.
Alternatively, is it more efficient to simply modify connection type within the hosts file / YAML playbook?

As documented here, ansible does not support telnet as connection plugin. This means you cannot use ansible to connect to a remote machine via telnet and execute any of the ansible modules.

However, you can use connection: local for local connection(run ansible against the machine that is running it) and send telnet commands from the machine where ansible runs.

Example playbook will look something like that:

---
- hosts: localhost
  connection: local
  gather_facts: false

  tasks:
    - name: my first telnet task
      telnet:
        username: user
        password: pass
        command:
          - my command


Octopus server installation on Linux I learnt that,

Octopus server can be installed on Windows 2008, 2012, 2016 server editions

I see msi packages on Download page

Can Octopus server be installed on Ubuntu?

Can tentacle software be installed on Linux box?

Historically Octopus Deploy was a solution to deploy software on Windows only, therefore the server component only supports installation on Windows.

This is for a couple of technical reasons:

Octopus Server takes a hard dependency on Internet Information Services (IIS) which itself only runs on Windows.
Octopus Server is written in the .NET Framework doesn't run on Linux.

Once you have the Octopus Deploy Server software installed, you can manage Linux hosts via SSH, you don't need tentacles to manage Linux hosts.


How to dimmension VMWare VM linux servers to run kubernetes container clusters? I'm designing a web apps and web services hosting platform using kubernetes clusters of VMs running docker containers (web services, web apps, databases).

This application hosting platform will be set on top of an existing VMWare infrastructure. We have 3 physical hosts powered by ESX.

Today a network of some 10s of VLANs runs approx 100 to 200 virtual servers with ~1 to 4 cores and 1 to 8 GB of RAM, with some 10s of GB of hard disks (classical HD, no-SSD).

Today our system is managed the old way, with named VM servers running one app or one database.

What I have to do is to create a new VLAN of some VMs (maybe 5 or 6 in a first time), running a kubernetes cluster to host web applications, web services and databases docker containers (developed and deployed by our gitlab on-premise instance).

Once this VLAN and kubernetes cluster will be up and running, tested for a while and validated by me and the direction, the project is to extend this architecture to the whole VMWare infrastructure, VLAN after VLAN.

The question is, how to dimension (vertical scaling) the underlying linux VMs composing the kubernetes cluster ?

How much cpus, RAM and disk space should I allocate by virtual host to be able to maximize the benefits of horizontal scaling allowed by the containerized architecture, the nginx load balancing and the mongodb sharding ?

There are several related system engineering trade-offs and choices involved in answering your question. It may be helpful going forward to tackle this question by first prioritizing your objectives and then making the selection and trade-off decisions about VLAN structure, cluster(pod) sizes, etc. to decide which options give you the most bang for the buck in vertical scaling.

For example, you didn't explain how cpu intensive your web apps are or how much load balancing (based on the front end API management and back end messaging) you expect you will need. While load balancers may not seem relevant to your Docker/K8s design question, the impact this can have on how you size your cluster nodes is obvious if you consider what impact too many nodes and a poorly distributed or overworked load balancer could have on your VLAN design.

The 'one app per VM' design you described suggests your app's workloads are pretty big (for an 8GB VM). Again need more info to delve into this more deeply, but assigning a DB VM per App VM seems a bit wasteful in terms of K8s worker/manager efficiency unless there's a severe reliability and resiliency requirement.

Next, what are your options to replace the magnetic HD with SSD? Costs for 100GB and 500GB SSD have dropped significantly and replacing your mag HD with SSD could certainly give your clusters a good deal of disk performance headroom when you scale up. In addition to performance, despite the added costs, overall system reliability (MTTF, MTTR) will increase dramatically. New SSD drives have 6GB data channels which is probably at least double what your mag HD have now. In building a Docker/K8s cluster of any size, disk channel bandwidth and processor core count can cover a multitude of design sins.

Finally, not sure what your project delivery timeframe is for this, but it might help your thinking all this over and making the best choices if you can take a step back and model this with SysML (using MS Visio) and look at your design and then apply Architectural Tradeoff Analysis Method.

ATAM and the theory behind it, despite its name, is really rather simple both in theory and in practice. ATAM gets its name because it not only
reveals how well an architecture satisfies particular quality goals (such as performance or "modifiability" or scaling in your case), but it also provides insight into how those  goals interact with each other — how they
trade off against each other. Doing ATAM upfront does point out critical design decisions with the most far-reaching consequences and are the most difficult to change after a system has been implemented.

SysML and ATAM is also a kind of insurance policy against painting yourself into a corner when you are asked to cost justify your design choices or forecast TCO when a change in either the web app design or your VMWare infrastructure occurs in the future.

ATAM info here >>>

SysML info here >>>


How to allow users to list authorized buckets in GCP Storage? I created a storage bucket B and I granted a user as "Storage Object Viewer" on it.
The user can view bucket content with a direct link (e.g. https://console.cloud.google.com/storage/browser/B/?project=azerty).

Is it possible to allow a user to list all buckets that he is allowed to read? It would be more user friendly to see them in Storage Browser Web console i.e. https://console.cloud.google.com/storage/browser

It's not currently possible to list all the accessible buckets across all cloud projects, neither in the developer console browser nor via the gsutil command. Both of them only display buckets for one project at a time.

But if you specify a particular project (or stick with the one selected by default) you can list the accessible buckets inside that project in both gsutil (as shown in @chupasaurus's answer) and in the developer console browser. In the browser you get to the project selection popup by clicking on the project's dropdown menu in the top navigation bar.

You could try to file a feature request for it at https://issuetracker.google.com/components/187243/manage#basic, it might get accepted if it shows enough interest.


Does Spinnaker's rosco use HashiCorp's packer or is it another tool? I am confused. Does Spinnaker use HashiCorp's Packer for baking or is it another tool? The reason for asking this that a folder called packer resides in the github repo of rosco https://github.com/spinnaker/rosco/blob/master/rosco-web/config/packer/docker.json

From spinnaker own documentation

  Rosco is the bakery. It is used to produce machine images (e.g. GCE
  images, AWS AMIs, Azure VM images). It presently wraps packer, but
  will shortly be expanded to support additional mechanisms for
  producing images.

So yes, rosco uses HashiCorp's packer.

The main reason to use packer is that you just need to change the provider used to create a docker image or an AWS ami or an Azure image and keep your building step identical.


declaratively mount a service account secret in OpenShift I am using helm charts to deploy some webhook handlers. The handlers need to connect to the cluster they are running on to deploy the configuration within it. It works but one step is tricky to move into the helm chart. When the chart makes the service account it gives it a randomly named secret:

$ oc describe sa sa-build-webhook-realworld
Name:                sa-build-webhook-realworld
Namespace:           your-eng2
Labels:              app=sa-build-webhook-realworld
Annotations:
Image pull secrets:  sa-build-webhook-realworld-dockercfg-4qz9g
Mountable secrets:   sa-build-webhook-realworld-token-bqtnw
                     sa-build-webhook-realworld-dockercfg-4qz9g
Tokens:              sa-build-webhook-realworld-token-bqtnw
                     sa-build-webhook-realworld-token-k7lq8
Events:              <none>

I can grab that and set it on the deployment config with:

#https://codereview.stackexchange.com/a/212095/75693
SECRET_NAME=$(
    oc describe sa sa-tag-realworld |
    awk -F': *' '
        $2  { KEY=$1 ; VALUE=$2;  }
        !$2 {          VALUE=$1; }
        KEY=="Mountable secrets" && VALUE !~ /docker/ { print VALUE }
    '
)
oc set volume dc/webhook-realworld \
  --add --name=sa-secret-volume \
  --mount-path=/sa-secret-volume \
  --secret-name=$SECRET_NAME

I am trying to see if there is a way to do this declaratively in the chart yaml that creates the dc and sa at the same time which is here.

Is there a way to reference the generated service account secret from the deployment config in the chart?

Or should I be trying to get helm/helmfile to do the command-line lookup as part of its work?

Okay the answer to this is that you can set spec.serviceAccountName on the dc and the secrets are mounted.

Annoyingly the same does not hold true for BuildConfig objects. They have an inconsistently named spec.serviceAccount. Setting that doesn’t mount any secrets associated with the sa. It also means your build isn’t an “image puller” until you explicitly set that up. So this configuration option appears pretty useless as at openshift 3.11.


How to prevent group from deploy to environment but allow same to redeploy? In our pipeline, the dev team will do multiple releases to their own test environment, when they are satisfied, they will queue a release for test. This signifies to our QA team that the devs are ready for QA to begin their testing.  QA approves the queued build, kicking off the automation process of deploying the artifacts into their environment.

Next, QA does their testing and puts in work items for bug that arise. The dev team then makes code changes and repeat the cycle until QA approves a release.  The release will then go to 3rd party compliance testing before being packaged up to go onto remote devices.

We would like to prevent the QA team from deploying to test when a build hasn't yet been queued by the dev team. However, we want our QA to be able to re-deploy any build that the dev team had previously queued, without requiring the dev team to queue the build.

This is all in the VSTS release environment.

You can add a post deployment aprroval to the dev environment that the dev team has to approve.
And then you can add a pre deployment approval to the QA environment that the QA team has to approve.

Then a release will be automatically triggered for the QA environment after the dev team has approved the post deployment approval on the dev environment. But it will not deploy to the QA environment until the QA team has approved the pre-deployment approval.

Additionally, the QA team can re-release any previous release to the QA environment at will by going back to the release and clicking redeploy.


How to get Memory Usage from Windows Server 2012 R2 in OpenStack I have an Rocky OpenStack cluster, I want to monitor memory usage of instances.

monitoring of Linux instances is OK but Windows Server instances do not provide any data for memory usage.

I've searched a lot, I've installed balloon driver on windows instances but not worked for me. All components are updated to latest version.

So how can I get memory usage data from them?

In PowerShell

Get-WmiObject -Class win32_operatingsystem -Property TotalVisibleMemorySize,FreePhysicalMemory

Or

wmic os get freephysicalmemory

Or

systeminfo


Jenkins Git Client Plugin Issue Git Client plugin previously was version was version:3.0.0-rc(beta) which has security vulnerabilities so upgraded to latest version:2.8.6 (latest). But unfortunately all the dependency plugins were still looking for 3.0.0 and believing that 2.8.6 is older version (which was a bug in plugin) are still using older version instead of newer version which caused issue during the upgrade. So tried to revert back but there is not revert back to beta version from 2.8.6 to 3.0.0.

No Option for revert back to older version: And there is no such version in https://updates.jenkins-ci.org/download/plugins/git-client/ for manual upgrade of Git Client as before one was beta version.

Note this problem I faced in my project. So if someone else is also facing similar issue this might help them to resolve.

Solutions:

You have a few choices to resolve the security issue in git client plugin 3.0.0-rc.

1) Install git plugin 3.12.1 and git client plugin 2.8.5 from the Jenkins update center
2) Upload git plugin 3.12.1 and git client plugin 2.8.6 into your Jenkins server from the "Advanced" tab of the "Manage Plugins" page inside "Manage Jenkins".
3)Install git plugin 4.0.0-beta11 and git client plugin 3.0.0-beta11 from the Jenkins experimental update center.
4)Upload git plugin 4.0.0-beta11 and git client plugin 3.0.0-beta11 into your Jenkins server from the "Advanced" tab of the "Manage Plugins" page inside "Manage Jenkins"

If you choose git plugin 3.12.1, you are choosing the released git plugin and the most stable.

If you choose git plugin 4.0.0-beta11, you are choosing the most recent pre-release git plugin with more bug fixes than are available in git plugin 3.12.1.

Reference: Git Client Plugin

GitHub Link

Plugin manual download

Related Issue : https://issues.jenkins-ci.org/browse/JENKINS-59336


Terraform | Error "transport is closing" when "terraform apply" after upgrading to 0.12 Terraform v0.12.7
+ provider.aws v2.25.0
+ provider.null v2.1.2
+ provider.template v2.1.2

Hello, when I try run terraform apply after upgrading to 0.12 I get error:

module.elasticsearch.module.ec2.aws_instance.this[0]: Modifying... [id=i-075090c94e5117ce3]
module.elasticsearch.module.ec2.aws_instance.this[1]: Modifying... [id=i-01fddbae2487e2573]

Error: rpc error: code = Unavailable desc = transport is closing

Error: rpc error: code = Unavailable desc = transport is closing

Releasing state lock. This may take a few moments...

Also I found next errors in logs:

[DEBUG] plugin: plugin process exited: path=/home/user/env-beta/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.25.0_x4 pid=25405 error="exit status 2"
2019/08/29 14:47:18 [DEBUG] module.elasticsearch.module.ec2.aws_instance.this[1]: apply errored, but we're indicating that via the Error
pointer rather than returning it: rpc error: code = Canceled desc = context canceled
2019/08/29 14:47:18 [DEBUG] module.elasticsearch.module.ec2.aws_instance.this[0]: apply errored, but we're indicating that via the Error
pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2019/08/29 14:47:18 [ERROR] module.elasticsearch.module.ec2: eval: *terraform.EvalApplyPost, err: rpc error: code = Canceled desc = context canceled
2019/08/29 14:47:18 [ERROR] module.elasticsearch.module.ec2: eval: *terraform.EvalSequence, err: rpc error: code = Canceled desc = context canceled
2019/08/29 14:47:18 [ERROR] module.elasticsearch.module.ec2: eval: *terraform.EvalApplyPost, err: rpc error: code = Unavailable desc = transport is closing
2019/08/29 14:47:18 [ERROR] module.elasticsearch.module.ec2: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing

And I notice strange thing, terraform wants add to this ec2 credit_specification {} but instance type is i3.large...

Maybe someone can help me. Thanks in advance.

It's a bug in terraform aws ec2 module, last working version is 2.5.0


Azure DevOps pipeline upload to FTP after create directory on remote server I developed a .net core console application and I want to deliver it to a FTP server.

I managed it in Azure Devops with FTP Upload task. All good here.

But before upload content to FTP, I want to make sure that remote directory exists, if not then create.

I put command mkdir test in FTP command but I got error that command is unknown and deployment is not done

How to achieve that ?

I'm just about positive that this Task is buggy and custom FTP Commands do not work at all. However, if the directory you specify in "Remote directory" does not exist it will be created so don't worry about doing a mkdir.


Cannot connect to server running on host from Dockerfile / image build I have a server running on localhost:3440 on the host machine.
In a Dockerfile I have:

RUN apt-get install -y netcat
RUN nc localhost 3440 | tar -x -O > ores-$(date +%s.%N).tgz

I've read that using --net="host" with docker run will allow a running container to connect to a server running on the host. But my question is - when building an image, how can I connect to a server on localhost?

The error I get is simply:

  localhost [127.0.0.1] 3440 (?) : Connection refused

anyone know what I might be doing wrong?

First you have to start netcat listening in the background.
$ netcat -l 3440 &

Then build your Dockerfile.

#
# Ubuntu Dockerfile
#
# https://github.com/dockerfile/ubuntu
#

# Pull base image.
FROM ubuntu:14.04

# Install.
RUN \
  sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
  apt-get update && \
  apt-get -y upgrade && \
  apt-get install -y build-essential && \
  apt-get install -y software-properties-common && \
  apt-get install -y byobu curl git htop man unzip vim wget && \
  rm -rf /var/lib/apt/lists/*

# Set environment variables.
ENV HOME /root

# Define working directory.
WORKDIR /root
RUN apt-get install -y netcat
RUN nc localhost 3440 | tar -x -O > ores-$(date +%s.%N).tgz
# Define default command.
CMD ["bash"]


Keeping graphs of terminated Kubernetes pods in Prometheus/Grafana I love using Prometheus/Grafana to monitor and tune resource usage on my Kubernetes clusters. When pods are terminated, however, the graphs are disposed of it seems.

I am looking for a solution to this problem.

For informed decisions on how to tune resource requests I need history to be kept.

Does anybody know how to keep graphs for deleted Kubernetes pods in Prometheus/Grafana.

I am currently using this deployment:
https://itnext.io/kubernetes-monitoring-with-prometheus-in-15-minutes-8e54d1de2e13

The metrics for deleted pods still exists on the Prometheus Time Series Database. The problem is that the Grafana dashboards imported from kubernetes-mixin create the pod list when loading the dashboards.

To change this in Grafana, open the Variables menu, then change $pod variable to refresh On Time Range Change:

Go back to the panels and change the time range (e.g., to "This Month"). The deleted pods for that period should be listed in the dropdown menu.


Docker: subnet clients can't ping other subnet Two containers are created with this Dockerfile:

FROM ubuntu:18.04

RUN apt update && apt upgrade -y && apt install -y vim iputils-ping net-tools tcpdump iproute* iptables traceroute

The were created with the following commands in a Ubuntu 18.04 machine:

sudo docker network create --internal --subnet=192.168.2.0/24 netR

sudo docker network create --internal --subnet=172.16.1.0/24 netB1

sudo docker run --name R1_ipsec --hostname R1 --privileged --cap-add=NET_ADMIN --net netR --ip="192.168.2.100" -dit image1
sudo docker run --name R2_ipsec --hostname R2 --privileged --cap-add=NET_ADMIN --net netR --ip="192.168.2.200" -dit image1

sudo docker network connect --ip="172.16.1.254" netR R1

sudo docker run --name PC1_ipsec --hostname PC1_ipsec --privileged --cap-add=NET_ADMIN --net netB1 --ip="172.16.1.2" -dit image1

sudo docker exec PC1 route delete default gw 172.16.1.1

sudo docker exec PC1 route add default gw 172.16.1.254 dev eth0

The problem: PC1 can't ping the R1's interface 192.168.1.100, even if R1 have net.ipv4.ip_forward = 1

Doker version: Docker version 18.06.1-ce, build e68fc7a

The problem was in this line:

sudo docker network create --internal --subnet=192.168.2.0/24 netR

The --internal option makes the network unreachable from other Docker networks.

The solution was to remove this option.


How do you hide sensitive information held in variables on a docker-compose file I am fairly new to DevOps. I have managed to build myself a CI/DI pipeline for my open source project.

As soon as I check-in some code into GitHub, Travis CI takes over, it builds a fresh docker image of my code, pushes it to DockerHub, then the after_success step within my .travis.yml connects to my VPS via SSH, and runs a script on there which basically does a docker-compose down followed by a docker-compose pull && docker compose up.

All seems to be working correctly. But I have an issue with my docker-compose.yml. Right now as you can see, it has all my environmental variables. Those are considered secret information. How do I hide them from the docker-compose.yml file whilst having everything still working as before?

UPDATE

I went and created myself a variables.env file in the same directory as the docker-compose.yml. Inside the .env file, I put my old variables, for example:

Database__DefaultConnectionString: "Server=database;Port=5432;Database=bejebeje_identity;User Id=postgres;Password=admin;"

But in Travis CI I get the following error:

  environment variable name 'Database__DefaultConnectionString: "Server' may not contains whitespace.

Use environment variables on your host that are read by your docker-compose.yml file.  There are a couple ways to accomplish this.  One way is to create a .env file on the host that contains your variables.  Docker Compose will read these variables and apply them to your configuration.

Alternatively, or in conjunction with a .env file, you can set environment variables in your shell.  These variables will be applied the same way as those read from a .env file and actually take precedence.

Read more here: https://docs.docker.com/compose/environment-variables/


call container B inside container A using docker networks I have two docker containers A and B that runs on the same user-defined network.
Container A has a port published, container B hasnt.
I do my requests on container's A published port, but container A needs to do requests on container B.
The requests to B inside A are mapped to 0.0.0.0:(container B needed port)/route.
These fails.

How could I call B inside A without getting and changing IP based on the network every time?

If you have a user-defined network set to bridge, you can access other containers by their container's name which is automatically as hostname for containers inside same network.

So from containerA you could do ping containerB without problems. In your case it would be changing the setting to containerB_name:(container B needed port)/route