How can I persuade developers on my team to embrace "You build it, you run it"? How can I persuade developers on my team to embrace "You build it, you run it"? By that, I have this quote from Werner Vogels in mind:

  Giving developers operational responsibilities has greatly enhanced
  the quality of the services, both from a customer and a technology
  point of view. The traditional model is that you take your software to
  the wall that separates development and operations, and throw it over
  and then forget about it. Not at Amazon. You build it, you run it.
  This brings developers into contact with the day-to-day operation of
  their software. It also brings them into day-to-day contact with the
  customer. This customer feedback loop is essential for improving the
  quality of the service.

I'm specifically thinking of a set of developers that:

Were hired into a developer role, with little/no mention of ops-related tasks.
Traditionally have "thrown code over the wall" to an ops team.
Traditionally have a 9-5 work schedule, and are actively hostile to the idea of "pager duty", participating in disaster recovery, writing post-mortems, etc, especially outside of normal business hours. (Note: I only have very infrequent outages in mind for this; I am not proposing that we add after-hours customer support to this team's workload.)
Are not currently responsible for writing/supporting monitoring or alerting on their applications.

Suppose there is a team that is rapidly developing new cloud micro-services with a profile that is getting to be such that handing these services off to an ops team is sub-optimal because they can't keep up in regards to gaining deep knowledge of the services that is required to effectively manage and monitor them.  "You build it, you run it" would work better for this team because tasks could delegated to each responsible team member. So this team would begin taking part in designing infrastructure, monitoring/alerting tools for the services, and (very infrequently) responding to outage events.

I am specifically interested in methodologies, backed up by real world examples.  How this been successfully implemented in other workplaces, and if there any canonical steps to follow while implementing this?  Any links to write-ups that can support answers would be very helpful.

I think the easiest way is to change their performance goals so they are based off reliability as well as delivering code. Sell it as the company cannot succeed without both so why should the developers only be measured on one? The best way for them then to meet their performance goals is to be involved in operations.

Ultimately you need to convince them this is the best way for the company to succeed and therefore for them to succeed. That's hard and you can't expect them to be onboard from the start. They need to be sold on the value too.


How to run/deploy devilbox (Docker image) docker-compose on GCP without a Dockerfile? Devilbox comes with a docker-compose file to run a docker container:
https://github.com/cytopia/devilbox

However, on the GCP Kubernetes Engine it is only possible to start up a new container via a Dockerfile. I searched for a translation/conversion tool of the docker-compose file to a Dockerfile. I could not find an easy way and also read that this is not the go to approach. Is there a way to let it run in the Kubernetes Engine or do I have to create a manual instance, upload and run the code manually?
Any other approaches?

Firstly, docker-compose is is a tool for defining and running multi-container Docker applications. (https://docs.docker.com/compose/).
Dockerfile is single-container, if you want to run multi-containers on GKE you should convert docker-compose to kubernetes deployment.

You can manually convert or use http://kompose.io/ to convert automatically.


Using volumes on AWS fargate I'm new to fargate and even dockers. But I have used dockers for a couple of months. Only now, I am using fargate.
I create a particular docker instance as follows:
docker run -v localHostFolder/configFile:dockerFolder/configFile -v my-vol:/app --restart always --entrypoint "myEntryScript.sh" myDockerImage

This works well on dockers. What I do here is use a config file from my machine "localHostFolder/configFile" and load it onto the docker instance I create. I want the docker instance to be populated with some config values.
In addition, I use my-vol:/app to save state data from my docker container so if the container restarts, this data can be used.
Now, a few questions -

I understand Fargate gives u access to just the container and not the underlying host. If so, how do you accomplish the above?

When I try to create a fargate container, I see an option for volumes, but not sure how it works. In fact I see that the mount-volumes drop down is always empty and don't see an option to create one either. Is this something only an admin can do?

Can we do the above using Fargate at all and if so how? If not using volumes, is there an alternate way to do this?

I also noticed there's something called Service discovery that I think can be used to create a fixed dns record so other aws containers can use the dns host name instead of the IP. Is this possible with Fargate? I see it is doable with ECS, but not tried though.

Yes, I opened a support ticket and the AWS representative said that
Fargate is really intended as a stateless container solution. I took
that to mean it's more of a self-made Lambda rather than a managed
Docker solution. The suggestion to add persistent storage was taken
under consideration and will be added to the feature request list.
You can add the storage also at the time of container configuration may be you can use NFS or you can upload data from fargate to s3.

With bind mounts, a file or directory on the host machine is mounted into a container. Bind mount host volumes are supported when using either the EC2 or Fargate launch types. Fargate tasks only support nonpersistent storage volumes, so the host and sourcePath fields are not supported. For more information, see fargate task storage.

Your Amazon ECS service can optionally be configured to use Amazon ECS Service Discovery. Service discovery uses AWS Cloud Map API actions to manage HTTP and DNS namespaces for your Amazon ECS services. To create service discovery you can refer to here.


Limit throughput between Jenkins and Jenkins Agent I am trying to find a way to throttle the thruput on the request of our Network folks. They want me to not exceed 5.5Mb/sec. Is there a way to do this within Jenkins. Looking at the config files under
Manage Jenkins > Nodes > "Node name"
I couldn't find anything that would do this.
Also, googling for a solution only resulted in simultaneous build info.
Thanks

hmmm, tuff one.  dont think it's possible do do this directly
counter question, why do you wish to limit traffic ?  If you could do this, say through a network config of some kind, it owuld negatively impact build time.
From experience things that have an impact on traffic between agent and sever :

Artifacts: if you are storing artifacts within your builds they will be stored on the server. So at these points in your build process you will upload them from the agent to the server and depending on artifact size it will generate traffic.
Sync: Depending on what source control you use (if any) and how you use it it's possible that the sync to occur on the master and then transfered over to the agent.  You can probably change this so the sync only occur on the agent which will pretty much eliminate this part of the traffic between them.
Logs: all logs generated on the your jobs will be transfered back to the server.  Though to max out a connection here you'dhave to generate a whole lot of it !

It's posible some plugins can generate their own traffic.  Without knowing more on your tasks it's difficult to go further.
As to the actual question my first reflex here would be to look for ways to achieve this through a firewall or some other network system to throttle traffic.  Some will allow to target a specific application in addition to source/target IPs in their rules.


Load Balancer and EC2 in same subnet? I have a custom VPC with 3 public subnets and 3 private subnets. I have three ec2 instances with elastic IP addresses in the public subnets taking traffic on port 80 (security group is configured to allow access on port 80 in public subnets which are behind an Internet Gateway). I created an Auto Scaling Group and linked it with the three public subnets to increase/decrease instances dependent on traffic. I attached a Target Group to the Auto Scaling Group with access to port 80. Now I plan to create an Application Load Balancer and attach that Target Group to it so that the Load Balancer can distribute load using the Auto Scaler.

In the AWS docs 'Application Load Balancer Getting Started', it states the following:

Before you begin: Decide which two Availability Zones you will use for your EC2 instances. Configure your virtual private cloud (VPC) with at least one public subnet in each of these Availability Zones. These public subnets are used to configure the load balancer. You can launch your EC2 instances in other subnets of these Availability Zones instead.

Notice at the end of the paragraph it says 'instead', as if you should not have EC2 instances configured in the same public subnets as your Load Balancer. Am I interpreting this correctly? You should not have EC2 instances and Load Balancers in the same subnet?

The best practice is to have your web, app and db servers in private subnets and only the actual front end - which is ALB in your case - in the public subnet. The reason is that web, app and db servers shouldn't be accessible from outside even if by accident (e.g. through security group left too open).

Technically it will still work to have both the ALB and the instances in public subnet but it's not recommended for the above reason. Also make sure to lock down access to the instance port 80 and 443 only from the ALB SG if you don't want any direct traffic.

Hope that helps :)


Robuster and more practical solution is an encrypted file. For example

shell> cat vault1.yml
my_email: admin@example.com
my_tag: tag123

shell> ansible-vault encrypt vault1.yml
Encryption successful

shell> cat vault1.yml
$ANSIBLE_VAULT;1.1;AES256
33376431383930313965356364356136306338383238303032363165633962636366663939373237
3433383262306631643431346236653534316331643466660a326562663633346662656233353733
63373561636432653535666437656537326635363935366261666237353136313939323535336665
3333666466656664610a633430313138393238653065623231393165383162656262646139353730
63653231363465643237666465613631646539366262656537323932346530386364353132326234
3230623232393630396361333462343862323231323733376665

Then the encryped file can be used in various places. For example

shell> cat playbook.yml
- hosts: localhost
  tasks:
    - include_vars: vault1.yml
    - debug:
        msg: "email: {{ my_email }} tag: {{ my_tag }}"

gives

    "msg": "email: admin@example.com tag: tag123"

Including vault-encrypted variables in a task

It's possible to limit the scope of the variables to a task. This limits the exposure of the sensitive variables and makes the solution safer. For example

    - debug:
        msg: "email: {{ my_vault.my_email }} tag: {{ my_vault.my_tag }}"
      vars:
        my_vault: "{{ lookup('file', 'vault1.yml')|from_yaml }}"

gives the same result.


Preinstall packages for a nodejs Docker Image according to package.json I would like to prepare a Docker image for a nodejs project. The developers use package.json to list their node module dependencies. I would like to prepare the Docker image (as base on the Dockerfile), with most of the node_modules already installed "globally" on the image.

Is there a best practice to prepare nodejs images?

Depend of life cycle of you app, because Node libraries and dependences change or end life to obsolet, so if you app constant change version to deploy, you can use docker tag for describe version or time when it build image, is a good practice, so Dockerfile shall copy package.json, and too look when you need change image source OS, by example change ubuntu linux to alpine , many dependences or how to install "npm" and dependences for node, a example:

  FROM alpine:3.8

# Update
RUN apk add --update nodejs nodejs-npm

# Install app dependencies
COPY package.json /src/package.json

RUN cd /src; npm install

# Bundle app source
COPY . /src

EXPOSE 5000
CMD ["node","src/index.js"]

well , hope it help you


How to upgrade job-dsl files to Jenkins 2 pipelines? Is there a common practice for migrating a large repository of Jenkins job-dsl to Jenkins2 pipelines? I am looking for ways to test the behaviour of builds, that go a little further than comparison of generated artifact from same code base.

Actually we have a quite large infrastructure, deployed via Ansible, and we got a verification environment which is topologically equivalent to production, so that would not be a problem. I am more interested in validation practices we could automate, as we build more from than 300 project repositories, there are a few thousand jobs spread across a few Jenkins masters.

No, there still isn't any common practice for this since this is not very common.

I would recommend you to create a Development Jenkins Environment and run tests against it. Right now you will have to do that from scratch since there are not many open source repositories that could help you.

So, how does this Jenkins Development Environment look like?

For me it is a docker-compose file that you can run and get everything you need. If you can automate the creation of the jobs it will be a plus.


What are further known DevOps operation models? It seems that many people somehow try to define the software development process using a DevOps tools model but there is no alignment; still I assume that this is more a question of maturity/alignment, not of matter of taste.
Question: What are further known DevOps operation models?
Some examples I've seen at available popular sources:

Plan, Develop, Test, Release, Operate

Plan, Code, Build, Test, Release, Deploy, Operate, Monitor

Requirements, Development, Build, Testing, Deployment, Execution

Here is a quote from the Wikipedia page for the DevOps toolchain:

A DevOps toolchain is a set or combination of tools that aid in the delivery, development, and management of applications throughout the software development lifecycle, as coordinated by an organisation that uses DevOps practices.
Generally, DevOps tools fit into one or more activities, which supports specific DevOps initiatives: Plan, Create, Verify, Preprod, Release, Configure, and Monitor.
Another proposed toolchain model was referenced in 2014 as coming from "an unknown source" as Plan, Code, Build, Test, Release, Deploy, Operate and Monitor.

However, that same Wikipedia page also contains an image which looks like so:

This image, IMO, does not correlate with the text on that page.
DevOps Toolchain stages on Wikipedia, By Kharnagy - Own work, CC BY-SA 4.0Wikipedia
Sources:
https://devops.com/devops-2017-cdos-care/

Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how part of such toolchain looks like in a mainframe environment. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).

Your example about "Plan, Code, Build, Test, Release, Deploy, Operate, Monitor" seems to be missing these crucial phases:

Enforce predictable problems in production to be addressed before they actually have an impact in production (sounds like sales talk, no?). You might think this is just "impact analysis", but that is not it:

Impact analysis is what you do in the plan-phase (we all know that, right?).
Audit (= name of the phase we use for it), is around the time that the developer thinks/claims to be ready. Something like "unless there is anything else that may have happened in production while I was working on my change, I'm done and recommend to move forward". Think of things like "version regression issues". Eg because last night a fix was applied to production and you didn't retrofit that fix, so if you move forward with your change, you'll wipe out the fix again ... bad luck if you do!

Give me a red button, so that if anything else fails (and production is impacted), I just have to push a magic button, be patient (for a few secs) and be 100% sure a rollback completed with 0 rebuilds of any kind (= Backout is the name of the phase we use for it).

This boils down to "component level backups" of any files/components you will be updating by your change in production, whereas these backups are made at the very moment the change is going to be applied. and this for any type of artifacts included in the change.

If later on (5 mins, 2 days or 1 week later) something goes wrong in production, and it turns out to be caused by your change, all you have to do is to have an automated procedure (the red button!) in place that you can launch, and which simply restores the backup you should have available.

Attention, if there were database updates involved, this backout may not be an option anymore. While things like "backups overlaid" (because of concurrent changes/fixes got applied to production) is another scenario where the red button is no longer available.

Easy, no?

PS: I leave it to everybody's own judgement if my answer is yes or no DevOps-compliant.


Dockerizing a tool to test web apps? I'm new to docker/Jenkins and would like a hand understanding something I'd like to practice.

I'm trying to set up a tool (like owasp-zap) and using it to run against a container built of an intentionally vulnerable website (like owasp-juice shop). Then saving those results. My only experience in Docker is grabbing an existing image and running the container locally. I'm struggling to find docs about this and could use some guidance.

You can dockerize/containerize pretty much anything fairly easily.  You basically just have to add a Dockerfile that:

Inherits from some base image;  this provides the basic OS features and whatever else you need.  E.g. a Java app may inherit from alpine-jdk-8 (a slim linux type image) that comes pre-bundled with the Java Development Kit.
Copies in your code and any dependencies.
Sets any variables you require.
Exposes any ports your app needs (e.g. ports 80/443 for serving a web page).
Mounts any volumes to save permanent data.  Generally anything in a container is ephemeral;  it will be gone when the container dies.  So, if you want to save/export results permanently you can mount a volume for, say, your local computer's file system, or for an amazon s3 bucket, or whatever.
Specifies how to start the app (the entry point).

Here's an example for arbitrary python scripts; but it's the same deal for any other language/tool:

https://runnable.com/docker/python/dockerize-your-python-application

If you're taking an existing executable which may  be your case, you just do the same thing but instead of loading in your code, you load in the binary and set that as your entry point.

The container lives until the entry point program ends.  So, some containers can be short lived - like just executing a CURL or something.  Some are permanent until they  die/are killed (e.g. a web service).

Again, to save state/results, unless you're writing them to a database or some other external system, you will want to mount a volume.

You build your Dockerfile into an image with a docker build command; then it will show up in your local docker image registry (docker image ls -a).  After that, you can run it with a standard docker run command (where you will likely want to expose any ports/etc that are needed).  I assume you're good with this one since you already run existing images in docker.

Generally you will need to store your image in a registry so you can use it on/from other systems.  But if running it locally is all you need you're done now :).

You can store your own containers in DockerHub or in Amazon AWS ECR or in any other number of options if you require a registry.


Get Ansible to print lines for include_role(s) I figured this would be trivial, but can't find the answer anywhere.

Let's say I have a playbook.yml that looks like:

- hosts: all

  tasks:
  - name: I'm a block!
    block:
    - name: I'm a role!
      include_role: name=some_role

And then I have a main.yml in roles/some_role/tasks that looks like:

- name: I'm a task!
  git: repo=https://doesnexist.com/nope.git dest={{ install_path }}

What I would expect / like to be printed is:

BLOCK [I'm a block!] *********
ROLE [I'm a role!] *********
TASK [some_role : I'm a task!] *********
ok: [MACHINE]

Or something like that. Instead, what gets printed is:

TASK [some_role : I'm a task!] *********
ok: [MACHINE]

So why did I bother putting those names on the block and include_role? Is there any way to get what I want, or at least something close to it? The debug module doesn't do what I want, since it is very poorly formatted.

I was able to create a custom callback plugin. I started with the default callback code and modified it:

    def v2_playbook_on_task_start(self, task, is_conditional):

        if(task._role is not None and task._role != self._current_role):
            self._current_role = task._role
            self._display.banner("ROLE [" + self._current_role.get_name() + "]", color=C.COLOR_DEBUG)

        self._task_start(task, prefix='TASK')

Now the output looks like:

ROLE [some_role] *****************************************************

TASK [some_role : I'm a task!] ***************************************
ok: [MACHINE]

TASK [some_role : Some other task!] **********************************
ok: [MACHINE]

ROLE [another_role] **************************************************

TASK [some_role : I'm a task!] ***************************************
ok: [MACHINE]

With the role line being bolded. It's a work in progress, but it at least makes it a lot more obvious where roles are being executed. My determination on block is that it is impossible to do what I wanted, due to the way the block class is implemented in Ansible.


How to find the total ips allowed by an AWS subnet? Is there a way to find how many total ip addresses that an AWS subnet can hold?
If my subnet CIDR is a.b.c.d/n, then the total ips allowed in this subnet - are they decided by a, b, c & d as well?

AWS docs show what the reserved IP's in each subnet are: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html
The gist of it is, the first 4 IP's and the last IP of the subnet are off limits if your subnet starts with the same IP as the VPC.  It's probably a good rule of thumb to avoid assuming these IPs are going to be available in any subnet even though potentially the second subnet address reserved for DNS could be available for consumption in another subnet. A better rule of thumb to avoid under-sizing subnets in AWS. Like, never plan on using a subnet smaller than /24. You need them for auto-scaling groups, NAT gateways, VPN's, and networking hooks into other apps like DynamoDB or Lambda.


Is Hashicorp's Vault suitable for personal projects as well? The Vault documentation indicates that there is a master key and that multiple keys are required to access the secrets, but does this mean that Vault is not suitable for personal projects if there is only one PC?

Vault will distribute multiple unseal keys but there's nothing to stop you from keeping all of the keys in one place for a personal project.

The one time I did it I set the keys to 3 and kept them in different physical locations along with online locations.


How to get environment variables in ECS container? What AWS CLI command should I be using to access the environment variables in an AWS ECS cluster's task's container?

I've seen describe-cluster, list-tasks, describe-tasks, describe-container-instances, etc... I can't seem to find one that lets me look at environment variables.

Is this doable via the AWS CLI?

Environment variables are defined inside the container and some are passed in the task definition. So you can use describe-task-definition to see the extra variables.

To get the default environment of the container image, you can register-task-definition for task with command 'env' and the specific container image for which you want to find out and then run-task this task.


How to set up AWS Trusted Advisor alert for Service Limit warnings? All I want in life is for AWS to shoot me an email when any of Trusted Advisor's Service Limits go to WARN or ERROR status. I've checked their guides, tried chatting with support and no one can get that to happen, so I'm reaching out here.
Use Case
We spin up a lot of EC2 instances and CloudFormation stacks and everything uses lots of resources. Somehow we always hit a limit we didn't know about in the middle of a validation sprint and end up blocked for an hour while AWS takes their sweet time bumping our limits. I'd like to set up some kind of event to email devops when one of the Service Limits is close to being reached. I am alright with it checking as infrequently as once a week, but would prefer once a day if that kind of granularity is possible.
Attempts
I tried setting up a CloudWatch Event, but the only option I could get was to email us when the Service Limits were refreshed - which is useless, because then you're already in there manually looking at it and there's no point in an email.
Options
Support told me there's no default functionality for this inside of AWS, so I'm looking for either custom event options, or I will also accept something leveraging Jenkins2 and the AWS CLI if that is a possible route. If Jenkins is the way to go it doesn't need to send an email - I'd probably have it slack our devops channel, if that changes the answer any.

That's on my backlog but still on early stage, I just surfaced the idea, so here's are my findings so far:

Using Trusted Advisor as a Web Service

Mainly:
Poll a Trusted Advisor Check for Status Changes

This documentation example can be implemented with aws cli with the support subset of commands and targetting the 'us-east-1' region, you'll then have to filter on the region field in the TrustedAdvisorResourceDetail structure part of the list.

I still didn't had time to write a script to poll the API (I would aim at ruby with the AWS SDK per our usage more than a shell script to group the requests).

Hope this help getting started even if that's not really a complete answer.

Courtesy of Romeo Ninov: For a  recap of which options is available for which account type you can check this site


Startup of ECS Fargate task fails with ResourceInitializationError mount.nfs4 Connection timed out When starting up an Amazon ECS task with launch type FARGATE it fails with the following reason:

ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: command failed with 32: failed to execute command to invoke EFS utils: mount.nfs4:

Connection timed out

The fargate task is a docker image that needs to mount EFS as persistent storage.

Why is the connection timing out?

Underlying reason:

The ECS service's network access security group did not have permission to access EFS.

Solution:

Add an inbound rule for type NFS in the security group as described in this tutorial: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html#efs-security-group


Creativity use case scenario? i am a devops learner, just finished graduation actively looking for jobs in the field of DevOps.

I am learning many tools like GIT, Jenkins, Chef, Docker , Nagious, ELK, AWS, etc.

Can anyone link all the DevOps tools together and give me a scenario where and how a company uses these tools at different times?

Your question is quite vague/generic.

People store their source code (or generally, files) in git; let Jenkins run jobs (like CI/CD pipelines), use Docker to put their software into well-defined containers, Nagios to monitor their systems, ELK to collect, store and visualize (mostly logging, event) data, AWS to host their stuff, and Chef to manage their configurations.

If you are studying that and learning about it, then you should have gotten that kind of information, really. The scenario in real world companies is simply to use those tools for what they are made, in different combinations as needed.

If you really have just gotten some deficient information from your college or wherever you studied those things, then get a book like The DevOps Handbook and give it a read. It should clear up how all of this would work together in the real world.


Terraform: ignore_changes conditionally? (temporary EC2 instance public IP address) I've got a couple of EC2 instances managed by Terraform, with public IP addresses in a VPC.

With each Terraform run, I'm updating DNS records based on those public IPs.

I don't need these instances on all the time though, and when I shut them down, they of course lose their public IP address.

At first this was causing associate_public_ip_address to become false, and Terraform wanted to recreate the instances. lifecycle.ignore_changes solves this:

resource "aws_instance" "instance1" {
  associate_public_ip_address = true
  ...

  lifecycle {
    ignore_changes = ["associate_public_ip_address"]
  }
}

resource "cloudflare_record" "record1" {
  domain = "example.com"
  name   = "server1.example.com"
  value  = "${aws_instance.instance1.public_ip}"
  type   = "A"
}

But... the public_ip attribute also becomes blank ;). This causes an attempted update to the A record, which fails because you can't set a blank record.

So, I'm trying to work out a solution. I could use elastic IPs, which I might end up resorting to, but I'd ideally like to avoid that if I can. My next thought was... is there any way I could use ignore_changes conditionally, and thus ignore changes to the DNS record's value if it is blank?

Are there any other solutions you can recommend?

Maybe you could use "count" attribute for this?

resource "cloudflare_record" "record1" {
    count = ${aws_instance.instance1.public_ip == "" ? 0 : 1}"
    ...
}


Is there any other approach for tagging git branch using pipeline without credentials in pipeline code? I was trying to tag a git branch using Gitlab CI. With the help of this answer from this community,

How to tag source code using gitlabCI

I can successfully tag a git branch from Gitlab CI pipeline. But In that method, I noticed that we need to provide the gitlab credentials in that pipeline code. I know, we can pass that as Gitlab environment variables.
But I just need to know, is there any other method to do this operation? With a Key or Token or something rather than direct username and password?
Not sure whether my question legit? I was thinking about the security side of it.

I believe that what you need is just a Personal Access Token, or if you're a maintainer, possibly a Deploy Token. It's much safer than embedding user credentials.


Are there any truly simple artifact repositories? I'm looking for a binary-artifact repository, but everything I see is super complicated, to set up, use, and manage. All I'm looking for is:

upload repos as tgz/zip with metadata (name, date, tag)
query and download by SHA and/or name
a local tool (python-based?) to help create repos from a subdir with the proper metadata and signature

Even a web-based portal is almost more than what I need (sure, it would be nice.)
Doesn't seem like it should require all the infrastructure and complexity of archiva, artifactory, etc. Or maybe some of these tools have a simple mode and I just can't find it?

I generaly work with a CDN, you can upload files and compressed files, and many CDNs has api ,  like it, but exist many other not conventional solutions, by example insert code or script in a fake image using base 64 and upload it to server public , or use facebook pixel to add script or file. Hope this help you.


Datapoints motivating introduction of SRE in organisation As there is no Site Reliability Engineering dedicated stackexchange, I found this to be closes one.

There are multiple great resources to use as inspiration for slidedeck about SRE principles [SRE slides].

Still I can't find :

short
concise
examples
motivating spending resources to implement SRE in organisation.

Most what I experienced in my professional life were highly confidential cases and numbers. I am concerned that most numbers that SREs know, are to remain "internal" to be presented internally within corporations.

However, maybe you know some study, (preferably set of) nice examples of post-morthems (even one by one is good),  from which we could make a strong arguments like "after introducing SRE model into organisation velocity of changes grown from n to m release pushes per x, with increase of availability by y and decrease of costs by z" (brainstorming) or other hard data points?

[SRE slides] - some examples:

Site Reliability Engineering: An Enterprise Adoption Story (an ITSM Academy Webinar) by ITSM Academy, Inc.
SRE From Scratch by
Grier Johnson, Platform Engineer at Square
GOTO 2017 • Site Reliability Engineering at Google • Christof Leng

P.S. If this question could be rephrased to fit better into this site guidelines, please provide me a suggestion in comment and give me a change to improve. Otherwise, I will appreciate other better platforms (However e.g. reddit.com/r/sre did not make great impression to me)

The types of numbers you're looking for might be hard to come across, because they're highly variable (even within one organization, it varies service-to-service and team-to-team, in my experience.) The SRE Workbook is now available for free, and includes two case studies (chapter 3) that might be helpful. Also, New Relic's SRE eBook does a really good job of summarizing SRE in a concise way.

Another way to approach this would be to try to use what you know about your service today to create a risk assessment and estimate downtime you can prevent if you had SRE and dev support to eliminate those risks


Using SSH to open interactive session in ansible Is there a semi-easy way to use Ansible to open an interactive session?
I've spent a good amount of time making use of keys and other credentials stores to make a stellar ansible inventory file. Problem is, when I want to get on the servers that I can easily manage, I've gotta go out and find those creds and take the time to log into them.
It's almost easier for me just to run adhoc commands with ansible instead of logging in via ssh.  Is there a way to pause ansible with a running interactive session so I can get a command prompt?  I noticed that sometimes it lets me accept host keys, but other times it bombs out, (I think this happens when running ssh proxy), so I know things can be done interactively.

You could try ansinble-console

It mostly feels like you just have a shell open, unless you use and ansible task name as your command, in which case prefixing it with shell <whatever you wanted to run> should work.


AWS SNS-SQS: Checking a subscription already exists I've found it's a useful strategy to create relevant queues whenever a queue worker service boots. For instance, on service boot, it checks whether its queues exist already in AWS and creates them if they don't.
Something I'm struggling with now is how to ensure that an SNS subscription for the queue exists. Checking whether the queue exists and creating it if not is easy, subscribing a queue to a particular SNS topic on boot is also easy - but I can't seem to find a sensible way of checking whether a particular queue is already subscribed to a topic!
If it helps, this will be using the Ruby SDK - but I think that the clients across the languages are pretty much all the same. If I can find a sensible way with the cli client it will most likely be available in the ruby sdk.
The only thing I've come up with so far is to list all the subscriptions and then filter them, but this seems a really silly way of doing it as there could be tonnes of subscriptions - I'd rather not have to deal with paging responses on worker boot.
It would be great if there were a way to retrieve whether a subscription is active, based on parameters of the queue ARN and the topic ARN.

In the end I decided to go another way.
I don't think it was actually a good idea to modify things in that way when the worker boots. It can cause problems because if you have to do some emergency modification of the configuration those changes will be reverted the next time your worker instances restart.
Instead, on boot the worker checks if the queue exists. If the queue does not exist, it will create the queue and all its corresponding configuration including SNS subscriptions etc. It deliberately will only create the SNS subscription for the queue if also creating the queue.
This way, if some configuration has to be changed as an emergency in production, those changes won't be "reset" if the worker spins up new instances.
It is a bit annoying, but also safer I think.
Obviously, this is entirely personal opinion and what I went with in the end - but it seems to have served me relatively well so far. If anyone else has any better ideas or disagrees please do let me know.


Auto Deploy after push to git I have some Codeigniter projects on some domains (on the same server), and the project is already in git.

How to ensure server gets the new update when I push the code to git?

Can anyone explain to me step by step?

You can achieve this using Git and Jenkins job. Create a Jenkins Job to deploy your code to server and Setup Git webhook to Call Jenkins job.

Check this link: https://www.serverkaka.com/2019/02/configure-github-webhook-in-jenkins-job-pipeline.html

  OR

Create an empty Git repo on the server
Write a Git Hook to deploy the code
Deploy from the local computer

Check this link: https://medium.com/@francoisromain/vps-deploy-with-git-fea605f1303b


How can I simulate failure of an AWS Availability Zone? In order to test that our load balancers and Multi-AZ RDS instances are working the way I'd expect them to, I'd like to simulate failure of an AWS Availability Zone.

Short of locating one of the datacenters and setting fire to it (which probably wouldn't be successful anyway), how can I do this? (I'm aware of Netflix' Chaos Gorilla, but it doesn't seem to have been released to the public like Chaos Monkey has, and I'm not sure what techniques it uses to cause its chaos).

This (slightly old in cloud years) question on Hacker News provides a few suggestions:

Remove the security groups from all instances in one zone, which should technically make them all appear 'down'
Adjust the VPC route tables to black hole the subnets in one availability zone
Disable an availability zone from the load balancer (docs)

You could also probably adjust the Network ACL attached to the relevant subnets to deny all traffic.

I'm not aware of an existing, available tool that does any of this automatically, though.


How can I find out when Terraform will support newly released AWS services? For example, AWS announced their new "RDS Proxy" service at re:Invent 2019 and it is now in preview. How can I find out when/if Terraform will support this (for planning purposes)?

https://aws.amazon.com/about-aws/whats-new/2019/12/amazon-rds-proxy-available-in-preview/

Terraform is an open source project that is managed on GitHub. You can follow existing issues or open a new one if your enhancement request isn't found. See their issue list at: https://github.com/hashicorp/terraform/issues

As Martin Atkins points out in the comments, the plugin specific features are each developed and tracked in their respective repositories. Here's the one for AWS: https://github.com/terraform-providers/terraform-provider-aws


What is the best way to deploy a compiled language project with docker and jenkins? I have a java project, and I'm generating .jar file.
My question is what is the best practice to use docker and jenkins for build and deploy the project?
Is it build the jar with one docker image, then put the Jar file in another docker image and run a container from the last image in the production env or is there a better way to do that?

I do not know how how about Jenkins, but GitLab CI has feature called artifacts which allow you to

explitely share some files/directories between builds
download them from GitLab Pipeline view

Additional pro is that allows you to make deployments manual actions.

Also if you are building fat JARs you do not need Docker for your deployments at all.

So the best solution for you is to have separate task for building release and separate for deployment (which can be manual job or not).


How to explicitly kill the process with PID 1 from inside a container? For demonstration purposes, I would like to kill the process with PID 1 from inside a container. But apparently, kill 1 doesn't work:

$ sudo docker run -it centos
[root@3752d3a44f10 /]# ps
  PID TTY          TIME CMD
    1 pts/0    00:00:00 bash
   15 pts/0    00:00:00 ps
[root@3752d3a44f10 /]# kill 1
[root@3752d3a44f10 /]# kill -9 1
[root@3752d3a44f10 /]# kill -SEGV 1
[root@3752d3a44f10 /]# <-- shell is still running

Is there a way to explicitly kill the process with PID 1 from inside a container? How?

PID 1 is a special PID which will ignore certain signals unless handlers are explicitly created. Docker has some documentation on this:

  Note: A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. So, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.

From here: https://docs.docker.com/engine/reference/run/#foreground

To kill PID 1 you will have to explicitly declare the handler for the SIGTERM signal or, in current versions of Docker, pass the --init flag in the docker run command to instrument tini.


Why label k8s deployment? In the below deployment yml, the deployment itself has a label "app: hello". What would happen if I delete that line (line 6)?

I understand the functional need for pods to be labeled, because then the deployment can reference those pods by labels. In the below example, that means that I understand why lines 11-12 and 16-17 exist. But that doesn't answer why the deployment should be labeled.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloweb
  labels:
    app: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
      tier: web
  template:
    metadata:
      labels:
        app: hello
        tier: web
    spec:
      containers:
        - name: hello-app
          image: registry.gitlab.com/itential/cloud-team/applications/hello-app
          ports:
            - containerPort: 8080
      imagePullSecrets:
        - name: gitlab-reg

Labels lets you as user query for particular objects; from the docs:

  Labels enable users to map their own organizational structures onto system objects in a loosely coupled fashion, without requiring clients to store these mappings.

For example if you use some kind of deployment tool like Spinnaker; the tool labels every single object that it touched so it knows about it - can manage it.

When you query for a label you can use equality-based operators (=, == and !=). K8s doc should be good starting point, alternatively check out this article on labels.


Docker on Windows: Write directly to host's EventLog? I run all my services as Docker containers on Linux by default. Recently got an order from a very large company to deploy to their Windows Servers. They have a very specific request for all application logs to appear in the host machine's EventLog. Their administrators don't know much about Docker and they don't want a centralised logging solution, like Kibana, for this deployment.

Is there a feasible way to achieve that on Windows?

Cheers

EDIT

The services are all written in .NET Core 2.2.

On Windows (Dotnet Core) it can be done using powershell remote-invoke.

In the container you need to set a trusted host by doing:

RUN @powershell Set-Item WSMan:\localhost\Client\TrustedHosts -Value '*' -Force

Ideally replace the * with the host actual IP address. Keep in mind it changes between system restarts.

On the host you need to create an Event Source called MyLogger.

After that in your app you can create a custom logger that you add to the Microsoft Logging pipeline.

RemoteEventLogProvider.cs:

using System.Management.Automation;
using System.Net;
using Microsoft.Extensions.Logging;

namespace LogWriter
{
    public class RemoteEventLogProvider : ILoggerProvider
    {
        private readonly string _computer;
        private readonly PowerShell _powershell;
        private readonly NetworkCredential _credential;

        public RemoteEventLogProvider(string computer, NetworkCredential credential)
        {
            _computer = computer;
            _credential = credential;
            _powershell = PowerShell.Create();
        }

        public ILogger CreateLogger(string category)
        {
            return new RemoteEventLogger(_computer, _credential, _powershell, category);
        }

        public void Dispose()
        {
            _powershell.Dispose();
        }
    }
}

RemoteEventLogger.cs

using System;
using System.Management.Automation;
using System.Net;
using Microsoft.Extensions.Logging;

namespace LogWriter
{
    public class RemoteEventLogger : ILogger
    {
        private readonly string _computer;
        private readonly string _category;
        private readonly PowerShell _powershell;
        private readonly NetworkCredential _credential;

        public RemoteEventLogger(string computer, NetworkCredential credential, PowerShell powershell, string category)
        {
            _computer = computer;
            _category = category;
            _credential = credential;
            _powershell = powershell;
        }

        public IDisposable BeginScope<TState>(TState state)
        {
            throw new NotImplementedException();
        }

        public bool IsEnabled(LogLevel logLevel)
        {
            return true;
        }

        public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)
        {
            var script = ScriptBlock.Create($"$password = ConvertTo-SecureString '{_credential.Password}' -AsPlainText -Force;" +
                                            $"$credential = New-Object System.Management.Automation.PSCredential ('{_credential.UserName}', $password);" +
                                            $"Invoke-Command -ScriptBlock {{ Write-EventLog -EntryType {logLevel} -EventId 0 -Source MyLogger -LogName Application -Message \"{formatter(state, exception)}\" }} -ComputerName {_computer} -Credential $credential;");

            var results = _powershell.AddScript(script.ToString()).Invoke();

            _powershell.Commands.Clear();
        }
    }
}

Example usage Program.cs:

using System;
using System.Net;
using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;

namespace LogWriter
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var config = new ConfigurationBuilder()
                      .AddCommandLine(args)
                      .AddEnvironmentVariables()
                      .Build();

            var computer = config["ConnectionStrings:Host.Address"];
            var provider = new RemoteEventLogProvider(computer, new NetworkCredential($"{computer}\\username", "password"));

            using (var factory = new LoggerFactory().AddConsole())
            {
                factory.AddProvider(provider);

                var logger = factory.CreateLogger("MyLogger");

                logger.LogInformation($"Remote log at {computer}");

                while (true)
                {
                    try
                    {
                        logger.LogInformation($"The time is {DateTime.UtcNow}");

                        if (DateTime.UtcNow.Second % 2 == 0)
                        {
                            throw new Exception("random shit");
                        }
                        else
                        {
                            logger.LogWarning("Example warning");
                        }
                    }
                    catch (Exception ex)
                    {
                        logger.LogError(ex, ex.Message);
                    }

                    await Task.Delay(TimeSpan.FromSeconds(5));
                }
            }
        }
    }
}

Example Dockerfile:

# Build
FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app
COPY . /app

RUN dotnet publish -c Release -o /app/out

# Runtime
FROM mcr.microsoft.com/windows/servercore:ltsc2019
RUN curl -o c:\dotnet.exe https://download.visualstudio.microsoft.com/download/pr/a9bb6d52-5f3f-4f95-90c2-084c499e4e33/eba3019b555bb9327079a0b1142cc5b2/dotnet-hosting-2.2.6-win.exe
RUN c:\dotnet.exe /quiet /install

# Host
RUN @powershell Set-Item WSMan:\localhost\Client\TrustedHosts -Value '*' -Force

# Application
WORKDIR /app
COPY --from=build-env /app/out .

ENTRYPOINT dotnet LogWriter.dll


What is Chef Automate I've noticed that now most of the Supermarket's recipes are using Chef Delivery (aka Chef Automate). There is little information about it on their site, and it looks for me like it is a "pig in a poke": first you pay and they you know what you've paid for.

Does Chef Automate require you any additional server, besides Chef Server (we already have it). I guess yes.
Is it required at all to write cookbooks now?
Will it replace Foodcritic, ServerSpec and Kitchen eventually? Also, I see they now offer and "InSpec" which seems to be similar to ServerSpec.

For the whole points and to try things there's https://learn.chef.io which allow you to test automate also.

Mainly chef automate is the next iteration bringing together 4 commercial products from Chef: chef manage (UI), chef reporting, chef compliance and delivery (CI/CD) with the addition of push jobs which was open sourced a little before.

Inspec is another new tool for compliance, it is based on serverspec but not extending it not using its code base.

Automate is not mandatory to author cookbooks at all, it is just a convenient way to display back status of nodes either on configuration point of view or compliance point of view (see the audit cookbook for inspec usage within a run), you can still does everything on your nodes on the free open source model, you'll have to handle the feedback loop yourself, either through a handler or any other way of your choice as it was already the case.

The CI/CD part of Automate doesn't replace any tool, it's more another approach to CI/CD pipelines, so you may use it to do the usual linting (foodcritic/cookstyle), test (chefspec), and then upload to your chef-server for use. It can handle deploys through the push jobs feature and after review and approval of changes.

Presentation of automate and the related documentation (workflow is the first concept, others are accessible from the menu on the left)


Combining Two Containers I'm working on a Windows laptop and running Docker for Windows on it. I want to run an Ubuntu container with a specific version and a MySQL server on it. So the question is, do I have to download MySQL on the Ubuntu container or can I run 2 containers (Ubuntu and MySQL) and combine them? How do I combine these 2 containers?

At first, I misread your question, thinking you wanted to run more than one service together, so I had recommended Docker Compose which makes it easy to run multiple services together, with networking, and can even specify the start up order.
But now I see you are just trying to run MySQL, and wondering do you need separate containers for the OS and MySQL: in short, no you don't need two containers. Instead, just get the MySQL image from DockerHub at https://hub.docker.com/_/mysql?tab=description
The reason you can just use that MySQL Docker image by itself is because it already is installed on a Debian image, which you can see in the Dockerfile at https://github.com/docker-library/mysql/blob/master/8.0/Dockerfile where it shows:
FROM debian:buster-slim

So, to solve the problem, do:

docker pull mysql
docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag as mentioned in the instructions on the Docker Hub page. Additional documentation at https://github.com/docker-library/docs/tree/master/mysql

I do strongly recommend using Docker Compose (docker-compose command) to run containers, as it makes it easier to organise and run them.


How to synchronize deployments (especially of database object changes) on multiple environments I am the DevOps engineer and a software engineer in a team where months back, the developers moved from having a central Oracle DB to having the DB on a CentOS VM on their individual laptops. The move from a central DB was to reduce dependency on the DBAs and also to eliminate issues that stemmed from inconsistent data.

The plan for sharing and ensuring synchronization of the Database with everyone on the team was that each person will share change scripts with everyone. The problem is that we use Skype for communication (we just setup slack but are yet to start using it fully), and although people sometimes post the text of DB change scripts, it could be missed by some. The other problem is that some developers miss posting the changes. Further, new releases are deployed in Production without being deployed on the Test and Demo environments.

This has posed a serious challenge for us, especially myself who of recent, became responsible for ensuring that our Demo deployments were in sync with the Production deployments.
Most of the synchronization issues border on the lack of sync of the Database due to missing change scripts or missing DB objects. Oracle is our DB of preference.

A typical deployment in the Demo environment is a very painful process that involves testing an application and as issues occur due to missing DB table columns, functions, stored procs, we have to look for the missing DB objects, apply them to the DB and then continue until all issues are resolved.

How can I solve this problem to ensure smooth, painless and less time-consuming deployments? Can migrating our applications to Docker help with the DB synchronization issues and the associated lack of discipline of the developers? What process can we put into place to improve in this area?

I would integrate the schema management into the application itself (or along with it).

Any change to the schema should be committed along the application code (and hence tagged also).

There's already a bunch of possibilities listed in this question: What practices or tools enable Continuous Deployment of Databases

With this kind of tools, using an in memory database like h2 or a central database and having the application check the schema version (stored in the DB) when started allow to get an updated database schema at application startup.

This will not solve your discipline problem and may create new ones, as this usually comes with a strict versioning of the schema, enforcing the version can be done with pre-commit hooks but that's a hard work.


Using Anaconda instead of pip+virtualenv We are currently using pip+virtualenv and install all our Python dependencies during the application build step on the CI. This requires compiling some of the packages, which contain C modules, which takes time and requires the build servers to have required system-level "development libraries" installed.

From what I understand, Anaconda can install pre-built packages allowing to skip the complication step altogether making the builds faster and not needing to have certain development packages.

Has anyone gone through pip+virtualenv to conda transition? What could be the downsides and potential problems of switching to conda for building and installing Python dependencies?

In my field (science) Anaconda is probably the most common Python distribution in use. conda is the package manager (just to be clear of the distinction).

It is difficult to find fault in conda when compared with pip+virtualenv.

The one point that I would make is that the Anaconda distribution is meant to be installed per user. If you have a Python environment that is shared across users there is no way of doing the equivalent of pip install --user with conda (but read on for a surprising solution...). Nor users can create conda environments.

So, if you are considering a single user installation go ahead and try Anaconda. It is worth a try

Note that you can use pip with Anaconda... they are not incompatible. In fact that can be quite useful in a shared installation.

I have setup a shared-user conda installation where users can install user-space packages with... pip install --user.

In a shared installation I do not see how users can create their own conda environments (conda has environments). And I bet conda+virtualenv would be a mess.

Note that conda is not Python tied: First you can have a conda installation with both Python 2 and 3 - and conda manages environments very well. But you can also have conda install Perl, R. In fact that is quite common. Our Perl interpreter is provided by Anaconda ;)

Also I suspect conda is probably stronger in Linux and Mac OSes. Windows is less well supported. For other operating systems, there are probably no packages (or they will be limited).


What are the signs of an understaffed DevOps team? What are the typical signs and signals of a DevOps team being understaffed? How would you justify/explain a request for a new addition to a team?

I would love to keep the question generic, but here is some additional information:

We currently have 2 DevOps specialists working together as a team, but the demands and the amount and complexity of products are growing. We are thinking to request a new addition to the team, but having some difficulties explaining and proving why it would be a good idea.

There are four main reasons why you can feel your team is understaffed:

Poor organization and planning of work
Doing work someone else should be doing
Doing work that should not be done at all
Being actually understaffed

Start with a review of the first three points. Read the Phoenix Project on ideas how to do the first. Ask yourself for every task you help anyone with if it should be done at all and if it is you that should be doing the task or if you should simply enable whoever needs it done to do it themselves. This will give you some documentation on why all the work you do is necessary.

Next review the four types of work mentioned in the Phoenix project:

Business Projects - what you do for other teams in the organization
Internal Projects - what you do to make your work easier in future
Scheduled Maintenance - what you do to keep the lights on
Unplanned Interrupts - what you do because something went wrong

If the work of your team is sustainable, you will spend roughly the same amount of time on each of the four. If the unplanned work starts to creep up close to 50% of your time, it is a sign you are definitely understaffed.

You should be able to hire to stay about one person ahead of the unplanned work reaching 25% of your time, otherwise, one person leaving will send your entire team into a tailspin you might never recover from. Overprovisioning of people and technology have same reasons and benefits.


How to specify specific subnet in Terraform when using for each I am trying to create an ec2 instance that will hold my Jenkins server. I want this to be in a private subnet which I created using a for each loop. Below is my subnet for each loop and my ec2 instance resource. I have also included the error message. Idea is I want it in the first private subnet.

ec2 instance

resource "aws_instance" "jenkins" {
  ami           = "${var.ubuntuAMI}"
  instance_type = "t3.micro"
  availability_zone = "us-east-1a"
  key_name = "me"
  monitoring = true
  vpc_security_group_ids = [aws_security_group.ssh_access.id]
  disable_api_termination = true
  subnet_id = "${aws_subnet.private[each.key]}"

  tags = {
    Name = "Jenkins"
  }
}

subnet resource

resource "aws_subnet" "private" {
  for_each = var.subnet_numbers_private

  vpc_id            = aws_vpc.Main_VPC.id
  availability_zone = each.key
  cidr_block        = cidrsubnet(aws_vpc.Main_VPC.cidr_block, 8, each.value)
  tags = {
      Name = "Private-${each.key}"
  }
}

variable used by the subnet loop

variable "subnet_numbers_private" {
  description = "Map for private subnets"
  default     = {
    "us-east-1a" = 1
    "us-east-1b" = 2
    "us-east-1c" = 3
  }
}

error message seen when doing a terraform plan

The "each" object can be used only in "resource" blocks, and only when the
"for_each" argument is set.

When you want to refer to a specific instance of a resource that has multiple instances due to using for_each, you need to include the specific key of the instance you want in your references:

  subnet_id = aws_subnet.private["us-east-1a"].id

If you just want to select any one subnet from the list, without needing to specify its name directly, you can use an expression like the following to select a single key by sorting the keys lexically and taking the first one:

  subnet_id = aws_subnet.private[keys(aws_subnet.private)[0]].id

However, beware that if you do the above then any later changes to var.subnet_numbers_private that affect which subnet sorts first will cause a different subnet id to be selected, which will then in turn cause Terraform to plan to replace your instance to move it into a new subnet.

From the way you wrote the question it sounds like you want just one instance in a single subnet rather than one instance per subnet, but if you did want an instance per subnet then you can achieve that by putting for_each in the aws_instance resource too, using the subnet resource itself (represented in expressions as a map) as the repetition expression:

resource "aws_instance" "jenkins" {
  for_each = aws_subnet.private

  # ...
  subnet_id = each.value.id
  # ...
}

In this latter case, the aws_instance.jenkins instances will also be identified by the keys you selected for the subnets, so you'd have instances with addresses like aws_instance.jenkins["us-east-1a"] and aws_instance.jenkins["us-east-1b"] so both you and Terraform can see which instance belongs to which subnet.


How to add LinkedIn's Burrow to a Docker Compose? It appears LinkedIn doesn't have an official Burrow docker image on Docker Hub, but there are others who have forked it.

However, I can't find any examples of how to add any of them to a docker compose file that spins up ZK and Kafka something like this.

What am I missing?

This is the example I was looking for: https://github.com/linkedin/Burrow/blob/master/docker-compose.yml


Is is possible to configure settings in Bitbucket globally? It is possible to configure settings per repository in Bitbucket. Would it be possible to configure settings globally? There are more than 30 repositories at the moment and I would like to avoid that I have to configure the same settings per repository.

At the moment this seems not to be possible in bitbcuket at the moment. Therefore I have created an enhancement in the Bitbucket issue tracker.


Jenkins trigger build if shared library changes I have Jenkins Pipeline job define in my git repo as.

// File: deployment/jenkinsfiles/staging/Merge

@Library("my_shared_lib") _

import com.company.myteam.Constants

pipeline {
    agent { label "common" }

    triggers {
        pollSCM("H/2 * * * *")
    }

    options {
        buildDiscarder(logRotator(numToKeepStr: "50", artifactNumToKeepStr: "50"))
    }

    stages {
        stage ("Staging Merge") {
            steps {
                script {
                    def config = new Constants().repoconfig
                    ansiColor("xterm") {
                        myteam.pipelines.stagingMerge(config)
                    }
                }
            }
        }
    }
    post {
        always {
            cleanWs()
        }
    }
}

Then I created jenkins job configuration as below.

Now, When I check the Git Polling Log its polling shared-library too.

Started on Feb 25, 2020 5:33:07 PM
Using strategy: Default
[poll] Last Built Revision: Revision 681f996e689bdeb9cfca2b167002b8ccd99590cb (origin/staging)
using credential github_username_with_token
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Github Service Account Username with token
 > git ls-remote -h https://myuser@github.company.com/myteam/myrepo # timeout=10
Found 6 remote heads on https://myuser@github.company.com/myteam/myrepo
[poll] Latest remote head revision on refs/heads/staging is: 681f996e689bdeb9cfca2b167002b8ccd99590cb - already built by 53
Using strategy: Default
[poll] Last Built Revision: Revision e80628ec7c9dbc59decbc81a3b884dcaa963a8dc (refs/remotes/origin/master)
using credential github_username_with_token
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repositories
 > git config remote.origin.url https://myuser@github.company.com/myteam/jenkins-shared-libraries.git # timeout=10
Fetching upstream changes from https://myuser@github.company.com/myteam/jenkins-shared-libraries.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials Github Service Account Username with token
 > git fetch --tags --progress https://myuser@github.company.com/myteam/jenkins-shared-libraries.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Polling for changes in
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
 > git log --full-history --no-abbrev --format=raw -M -m --raw e80628ec7c9dbc59decbc81a3b884dcaa963a8dc..869b3a16e07d21a88d557f1857376bfed7717a6e # timeout=10

Whenever I change code in jenkins-shared-library, its start build for the job.

I tried workaround given in https://issues.jenkins.io/browse/JENKINS-39615?focusedCommentId=280729&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-280729, but no luck.

I changed my pollSCM to githubPush, but this also trigger when I change in shared-library.

I tried solution given in https://issues.jenkins-ci.org/browse/JENKINS-41497 as Ignore on push notifications for githubPush() but still its triggering job.

Before moving to pipeline, I had freestyle project. Where there is section Source Code Management, where we can define the branch, and githubPush() was listening on that branch only.

When we select project type as pipeline there is no Source code management section. May be due to that, its listening for all push events.

By push or poll, I want to restrict my jenkins job to trigger only when there is changes on myrepo.git only.

This issue can be solved by using legacy SCM option. Go to Jenkins -> Manage Jenkins -> Configure System and select "Legacy SCM" option for share libs. this will resolve the issue completely, It worked for us.


Ansible Playbook Localhost Invocation Working Directory I want to run an ansible playbook locally (host is localhost, like -i "localhost,") and inside the Playbook obtain the working directory from which the ansible-playbook command was invoked.

However, when I read out the current working directory in a playbook task, I get the directory of the playbook itself.

Is there any variable I can refer to to obtain the working directory on the controlling host (which is localhost) in my case?

I can inject via a variable -e but I would prefer to have a fall-back that if that variable is not set I can obtain it from the ([local]host) environment.

You can use env lookup plugins. Plugins are always evaluated in the context of "parent" ansible process on the control host.

- debug:
    msg: "{{ lookup('env', 'PWD') }}"

More lookup plugins can be found here: http://docs.ansible.com/ansible/latest/user_guide/playbooks_lookups.html


How I can use multiple hosts in AWX playbook? I have workflow that create 2 virtual machines on GCP. When virtual instance created I store output with external ip to variable. In next playbook i want to use all of my ip addresses in host section. My code is:

---
- name: Download and install jenkins
  hosts:
   - "{{ node_1_ip }}"
   - "{{ node_2_ip }}"
  tasks:
   - name: Install java
     yum:
       name: java-1.8.0-openjdk-devel
     become: true

But when I run my playbook i have warning:

[WARNING]: Could not match supplied host pattern, ignoring: 35.228.80.9

35.228.80.9 is my node_2_ip. How can I apply my playbook to this 2 machines?

I suspect you haven't set up the host in your inventory you can figure out where your hosts are being pulled from with the following command:

ansible-config dump | grep HOST

In practice, you probably just need to create an inventory file with the following content:

[nodes]
node01 ansible_host=IP OF YOUR FIRST NODE HERE!!!
node02 ansible_host=35.228.80.9

Then specify the -i parameter to ansible-playbook:

ansible-playbook -i inventory -k playbook.yml

Depending on your exact use cases it might be more scalable and maintainable to use GCE Dynamic Inventory, although this does require a little bit of extra setup.


Jenkins job using VPN? Is there a way to connect to a VPN for the duration of a Jenkins job? (freestyle or pipeline, I can work with either).

I scoured the plugin list figuring someone might have developed something, but the only page found was this OpenConnect page with no actual code developed, so that seems to be out.

We're running Jenkins on Ubuntu AWS boxes, if that changes anything. I'd prefer the ability to connect and disconnect in a single job, rather than making everything run on VPN.

I think that it's very difficult, if not impossible, to achieve something like this.

The reason is that once the machine running Jenkins is connected to the VPN it is also effectively disconnected from the original network it was connected to, which means Jenkins on that machine will loose connection with the rest of your Jenkins setup.

The only way that might work would be if using completely standalone one-machine Jenkins setups.


How do I run Terraform with AWS MFA? I'm using Terraform with the terraform-provider-aws provider to manage my AWS infrastructure. I'm trying to enable mfa_delete on an S3 bucket, but when I try to apply the change I get this error:

1 error occurred:
        * aws_s3_bucket.logs: 1 error occurred:
        * aws_s3_bucket.logs: Error putting S3 versioning: AccessDenied: Mfa Authentication must be used for this request
        status code: 403, request id: <redacted>, host id: <redacted>

How can I run Terraform in a mode that uses multi-factor authentication? My IAM account has a Virtual MFA device attached, but that MFA isn't used when I'm terraforming because terraform uses the API Access Key associated with my account.

The solution is to specify an assume_role statement:

provider "aws" {
  profile = "default"
  assume_role {
    role_arn = "arn:aws:iam::[ACCOUNT_ID]:role/terraform-test-role"
  }
}


How to ensure that the docker build fails if an ENV has not been set? This github discussion was found. It seems that it is allowed to build a docker image while a certain ENV does not have been set.

How to ensure that the docker build fails if an ENV has not been set?

An empty of non existent environment variable is the same thing, there's no reason Docker do not allow you to pass an empty variable as it could be a valid use case.

It's up to your responsibility to ensure needed environment variables are there and give an error message back when not.

Something along the line of:

RUN 'if [ -z "$AN_ENV_VAR" ]; then echo 'Environment variable AN_ENV_VAR must be specified. Exiting.'; exit 1; fi'

You can play with the exit code to automate some actions on failure around your docker build command.


Where is the git submodule revision actually stored? I understand how to use git submodule to reference a specific commit in another repo, but where is that reference actually stored? It doesn't seem to be in .gitmodules as plain text?

The .gitmodules file just records the path and remote URL to the submodule repository. The commit itself is stored in the tree object. You can query it like this:

git ls-tree <ref> <path>

where <ref> is a reference to some commit (e.g. HEAD, master, a commit SHA sum) and <path> is a path to a submodule.

The output looks something like this:

160000 commit <sha> <path>

where <sha> is the commit in the submodule. The number in front has the following meaning, according to the Git book:

  Notice the 160000 mode for the DbConnector (submodule path of the example used in the book) entry. That is a special mode in Git that basically means you’re recording a commit as a directory entry rather than a subdirectory or a file.

See also this question on StackOverflow.


Spot block vs reserved instances Let's say we have a continuous process where each two hours we receive some data which has to be analysed by a fleet of workers on AWS EC2 instances which always complete their job within two hours (let's say normally within 90 minutes).

What would be the most cost-effective solution in this case?

Use Spot blocks with finite duration of 2 hours?
Use fleet of reserved EC2 instances?

Though spot blocks seem to be less expensive, I am concerned about their availability as data has to be processed immediately after receiving within the next 2 hours.

Anything that "needs" to happen in a given time frame is not a candidate for spot instances, including spot blocks.

Spot capacity is, by definition, surplus/idle capacity, subject to availability, so your spot block may simply fail to start, as capacity may not be available.  It is less expensive for exactly that reason -- AWS is selling surplus capacity for which no one is currently willing to pay more.

Information about the pool of compute capacity that is currently flagged internally as eligible for participation in the spot market is not publicly accessible, and although it was possible to make some reasonable assumptions in the past about capacity based on signals you could obtain from the real-time pricing, the spot pricing algorithm was modified in 2017 in a way that largely masks those signals (and the change, in my opinion, was a good thing for any customer who was not aggressively gaming the system).

Spot blocks are intended to run for a committed period of time, so they are less likely to be interrupted than other spot instances, but they can initially start up successfully and then subsequently be interrupted at any time.

  In rare situations, Spot blocks may be interrupted due to Amazon EC2 capacity needs. In these cases, we provide a two-minute warning before we terminate an instance, and you are not charged for the terminated instances even if you used them.

  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#fixed-duration-spot-instances


Git Workflow with User Acceptance and Staging Environments I'm trying to streamline/standardize a git workflow in my workplace. The current delopment process is as follows:

Pull 'development' branch
Write code and commit and push to 'development' branch
Cherry pick from 'development' into 'testing' branch for User Acceptance Testing
Cherry pick from 'testing' into 'staging'
'staging' is merged into 'master' once a week

I want to adopt a 'feature branch' work flow with testing, staging and master. Thus I don't think there is any need for 'development'. From waht I gather, feature branches should be taken from 'master'.

What I'm having trouble understanding is how to fit in the 'testing' and 'staging' environments into a 'feature branch' git workflow.

Can someone explain how that might work?

My favourite workflow is the development-less "branch per feature" workflow from Adam Dymitruk (http://dymitruk.com/blog/2012/02/05/branch-per-feature/). It has these branches:

Exactly one master in an unbroken line from day 0 up to the version of your app that is currently in production.
Many feature-... branches.
One (or if you are so inclined more) qa branch.

The gory details can be found on Adam's site, but in short:

Every branch goes off of master.
All branches that are in active testing are merged into qa.
qa only ever contains merge commits (i.e. feature branches merged on top of each other, starting from master), never direct manipulations.
Whenever a feature branch gets new commits, and you want those in qa, you rebuild qa from scratch.
Whenever you want to deploy a bunch of features, you create a qa from all of them, test to your heart's content, and then simply rename that branch to the new master (which, as it started out from the old master, will be a direct, unbroken line of merge commits from day 0). Then, all feature branches are rebased on that.
One fundamentally important part for this workflow is to enable the git rerere cache feature, and distribute that cache between all developers. Hence, every merge conflict needs to be resolved only once, globally.

It's trivial to have more than one qa branch at a time to test different subsets of features; in your case, you could have qa (with everything nearly close to finished, i.e., compiles without error), qa-acceptance, qa-staging or whatever you like.

In my experience, there is always a relatively large main, ongoing qa branch (recreated from the master and features regularly), and when some features are picked for release, the will be a temporary smaller qa-....

It is trivial to add and remove individual features from the set of qa or release-candidate branches on the fly, as those are recreated constantly and only ephemeral.

You will find a lot of people very much objecting to git rebase, but if one wraps their mind around the fact that this is just a tool like any other, it will be fine. You need good tooling for this of course, but this is not complicated to program yourself; the actual git commands are all pretty simple and self-explanatory.

We have extended the workflow (or rather our tooling) to even allow arbitrary sub-features (i.e., feature branches going off of another feature branch, not master), so multiple branches can go off of a common ancestor which is not yet in production; works like a charm. But you can get pretty far without this.


Is there a book only about Ansible (without touching Vagrant, VirtualBox, Docker and virtualization stuff)? I need a book only about Ansible (without touching Vagrant, VirtualBox, Docker and other above-shell stuff).

About 6 months ago I tried reading the Ansible docs but these were totally un-understandable to me so I bought a good book by Jeff Geerling named "Ansible for DevOps" which is aimed of course for DevOps and thus includes extensive elaboration on software like Vagrant (OS-level virtualization-management tool), VirtualBox (OS-virtualization tool that is managed with Vagrant) and if I'm not wrong Jeff also touches Docker there, but this is not what I seek and had hard time understanding much because I had no DevOps background at the time besides some Bash.

I usually work with IaaS hosting services in which I can't even utilize Vagrant and VirtualBox and have no usage in Docker (I do well with Bash and some Bash extensions like Composer, Drush for Drupal and so forth which make Docker redundant for the projects I personally work on).

Is there a book that teaches Ansible but focuses only on Ansible, say, something that does general introduction to YAML, then to creating and playing Ansible playbooks on Linux environments (native/WSL)? Some kind of "Ansible for web developers" or "Ansible for general audience"?

I would highly recommend looking at this tutorial by digital ocean. How to install and configure Ansible on Ubuntu.

Although this tutorial is not a book, I had a similar situation with the ansible docs and trying to understand them when I first got started. This tutorial will provide an in depth hands on experience for getting started with ansible.


validation block iteratively This validation block works for a single input variable.

variable "mytestname" {

     validation {
        condition = length(regexall("test$", var.mytestname)) > 0
        error_message = "Should end in test"
     }
}

I need it to work inside a for_each - or have some workaround to accomplish this.
The issue is that there is a restriction on the condition statement - the condition HAS to take in the input variable itself (i.e. - it cannot accept an each.value)

variable "mytestnames" {

listnames = split(",",var.mytestnames)

for_each = var.listnames

     validation {
        condition = length(regexall("test$", each.value)) > 0
        error_message = "Should end in test"
      }
}

The above snippet does not work. I need a way I can iterate over a list of values and validate each of them using the validation block. There must be a way to do this without a validation block...??

It sounds like your goal is to have a variable that takes a list and to validate each of the elements of that list. If that's true, then the approach would be to write an expression that returns true only if all of the elements are valid. For example:

variable "names" {
  type = list(string)

  validation {
    condition     = can([for v in var.names : regex("test$", v)])
    error_message = "All names must end with \"test\"."
  }
}

The above relies on the fact that regex will fail if the given string doesn't match the given pattern, and that the for expression will fail if the given expression fails for any v. can then captures the failure and returns false, making the validation fail.


CircleCI manually approved Deployment with Rollback The new CircleCI 2.0 configuration allows to have a step of type approve which waits for user input. Is there some way to not just deploy code using manual stop, but also have a rollback to a previous version based on user input?

The answer, no.

The best ways to rollback would be to use git revert, which then adds the rollback to history (which is a good thing), and then push that like any other commit. The other would be to use the git hash of the commit you'd like to rollback to and build it with the CircleCI API.

Right now, you could also attempt to rebuild and older build via the UI but this isn't suggest for deploys & Workflows. Too many variables for safe, reproducible deploys.

- Ricardo N Feliciano
Developer Evangelist, CircleCI


Predictive autoscaling on K8s I am learning about using Machine Learning for provisioning proactive resources on the Kubernetes and I have found some of the interesting articles about it on the internet(for instance for Thoth or Microscaler). I have also read about predictive autoscaling in AWS and GCP however only for VM's, not to use in the K8s clusters.
https://docs.aws.amazon.com/autoscaling/plans/userguide/how-it-works.html
https://cloud.google.com/compute/docs/autoscaler/predictive-autoscaling#how_predictive_autoscaling_works
However I would like to understand more in-depth about the above solutions - how do they work internally and what is the possibility of moving that solutions to the K8s. Do You know where can I find informations about it? Or if that is even possible to find those information or they are hidden internally in those companies? Do You have any other scientific articles or solutions which are now available?
Thanks

These are presumably proprietary tools for EC2 and GCE respectively. The need for predictive VM autoscaling is pretty strong for Virtual Machines since provisioning time can be measured in the 10's of minutes depending on the configuration.
Conversely, application spin-up time in Kubernetes is typically measured in seconds. Predictive scaling for the Kubernetes HPA might have limited practical benefit.
Now using cloud provider predictive autoscaling to scale out the VMs running a Kubernetes cluster based upon ML could be pretty interesting.


How to implement a frozen test environment? Here is a partial quote from an answer to the question about "How to avoid continuous integration-caused instabilities in test environments?":

  This environment usually freezes during the tests.

My question: what are sample implementations of a frozen environment? I.e. what can you do to technically enforce that nobody (except if allowed by an authorized user like a release manager) will be able to change anything in such frozen environment.

Clarifications:

I'm not talking about what (I think) is called "frozen periods" during (eg) year-end processing in banks. That is about not being allowed to apply any (repeat any) changes to production environments, to reduce the risk of new changes/fixes being introduced that may impact the year-end processing.
Assume that users who are allowed to approve/apply changes anyway (such as the release manager in my example), will only do so in exceptional cases. Such as where during testing a high severity issue is encountered, for which deferring a fix to a next release is not an option (since it would production at risk if the release would be activated without such fix).
This could just be about suspending any automated update during the time of the test. The point is: avoid someone else upgrading an Application A to version Y while another team is still testing application B in version X which rely on application A. This could mean having a guard to avoid a test team to require an update on a dependency under test.

TeamCity has a Shared Resources build feature which allows you to define a resource which multiple Build Definitions depend upon it.  Build Definitions can either require a Read Lock or a Write Lock, you can also define if these locks are exclusive or allow a degree of parallelism.

If we make the following assumptions about a shared environment named PreProd:

A Shared Resource exists named "PreProd".
All build definitions, such as deployments, that make any change to that environment take an exclusive Write Lock on "PreProd".
All build definitions, such as non-restrictive tests, that perform read-only operations on the environment take a non-exclusive Read Lock on "PreProd".
TeamCity is the only process that can do anything on PreProd, albeit perhaps via another tool.

Therefore the following are true:

When a deployment is happening then nothing else can use PreProd, they will be queued.
When a test is running, any deployment would be queued until the tests complete.

You can use a similar mechanism with Jenkins using the Exclusion plugin.  In fact, you could build this functionality into any process using locking or semaphore - for example Apache ZooKeeper or HashiCorp Consul.


get value of default_statistics_target on postrgesql I want to see the value of default_statistics_target on postgresql before running

SET default_statistics_target=1000

Its most likely the default (100) but would like to see it.

Where it can  be found ?

Actually, there are two questions:

Where default(100) value is declared?
How can I see Stats target value before applying SET default_statistics_target=1000

Where default(100) value is declared?

You can see it in the source code

{
      {"default_statistics_target", PGC_USERSET, QUERY_TUNING_OTHER,
          gettext_noop("Sets the default statistics target."),
          gettext_noop("This applies to table columns that have not had a "
                       "column-specific target set via ALTER TABLE SET STATISTICS.")
      },
      &default_statistics_target,
      100, 1, 10000,
      NULL, NULL, NULL
  },

How can I see Stats target value before applying SET default_statistics_target=1000 ?

Let's suppose that column bar of table foo has  attstattarget=500:

alter table foo alter column bar set statistics 500;

So, there are two ways to show attstattarget

SQL_query:

SELECT attrelid::regclass, attname, attstattarget FROM pg_attribute WHERE attstattarget > 0 order by attstattarget desc;

 attrelid | attname | attstattarget
----------+---------+---------------
 foo      | bar     |           500

\d+ command:

\d+ foo

                                                            Table "public.foo"
          Column           |            Type             |                  Modifiers                   | Storage  | Stats target | Description
---------------------------+-----------------------------+----------------------------------------------+----------+--------------+-------------
 bar                       | varchar                     | not null                                     | plain    | 500          |

** Note**: Postgres doesn't show stats target until it distinct from default value.

300 - yet another magic number

Thx to this comment, you can read a paper about another magic number and check it in the source code:

/*
   * Determine which standard statistics algorithm to use
   */
  if (OidIsValid(eqopr) && OidIsValid(ltopr))
  {
      /* Seems to be a scalar datatype */
      stats->compute_stats = compute_scalar_stats;
      /*--------------------
       * The following choice of minrows is based on the paper
       * "Random sampling for histogram construction: how much is enough?"
       * by Surajit Chaudhuri, Rajeev Motwani and Vivek Narasayya, in
       * Proceedings of ACM SIGMOD International Conference on Management
       * of Data, 1998, Pages 436-447.  Their Corollary 1 to Theorem 5
       * says that for table size n, histogram size k, maximum relative
       * error in bin size f, and error probability gamma, the minimum
       * random sample size is
       *      r = 4 * k * ln(2*n/gamma) / f^2
       * Taking f = 0.5, gamma = 0.01, n = 10^6 rows, we obtain
       *      r = 305.82 * k
       * Note that because of the log function, the dependence on n is
       * quite weak; even at n = 10^12, a 300*k sample gives <= 0.66
       * bin size error with probability 0.99.  So there's no real need to
       * scale for n, which is a good thing because we don't necessarily
       * know it at this point.
       *--------------------
       */
      stats->minrows = 300 * attr->attstattarget;
  }
  else if (OidIsValid(eqopr))
  {
      /* We can still recognize distinct values */
      stats->compute_stats = compute_distinct_stats;
      /* Might as well use the same minrows as above */
      stats->minrows = 300 * attr->attstattarget;
  }
  else
  {
      /* Can't do much but the trivial stuff */
      stats->compute_stats = compute_trivial_stats;
      /* Might as well use the same minrows as above */
      stats->minrows = 300 * attr->attstattarget;
  }


Can we talk about "reproducible builds" when working with interpreted languages? This is a vocabulary question. I know the concept of reproducible build as the guarantee we can build software from the sources on a different host or at a different time, but producing an end-product identical at the byte level each time.

But  I'm not quite sure if we can talk about (or if there is an equivalent to) "reproducible builds" when working with interpreted languages (PHP, JavaScript, Python, ...)

Should I talk about "reproducible build" when it is just a matter of providing a well-controlled environment1?

1 throuh a virtual environement, a container, a VM or whatever elese

A "build" does not have though to be a compilation step.

".. the term build is similar to that of any other field. That is, the construction of something that has an observable and tangible result."
https://en.wikipedia.org/wiki/Software_build

More important in this context is to be able to reuse results of your build in other contexts to build new things.

That is, also in PHP and Javascript you could produce some packaged artefacts, be it a zip file or language specific archive linkable to a dependency manager: NPM in Javascript and Composer in PHP for example.


Ansible on Ubuntu Core Ubuntu Core is a minimalistic operating system. It uses snap package manager instead of apt.
Is it possible to manage a Ubuntu Core device with Ansible?

If the OS has a python installation, you can use  Ansible to manage it.. Note that you should check the individual module requirements (for example the snap module) since they may have dependencies which cannot be satisfied by your OS.

In general though the modules are written in python and as such a python runtime is needed.


How to visualize dynamic microservice architecture running with Consul? Using Consul, you could allow your teams decentrally deploy and run microservices with dynamic dependencies.

It is also possible to list the services: /catalog/service/:service

Now, how to assess these dependencies automatically if the architecture grows dynamically to get a visual representation of service inter-dependencies?

I recommend that you take a look at the NetworkX library for Python.  With a script you could query Consul and build a graph of linked nodes which could be rendered to an image file and served with a web server.  A Flask microservice for this would be fairly simple to set up and provide a very useful visualization of your microservices spiderweb.


How to remove a non-existing GitHub Ansible role from Galaxy? According to this documentation it is possible to remove a role from Galaxy, but the following fails:

ansible-galaxy login succeeded
ansible-galaxy delete 030 https://github.com/030/ansible-firewall resulted in:

  ERROR! Galaxy user 030 does not have access to repo 030/https://github.com/030/ansible-firewall

You have to pass only the repository not the full HTTP URL of the repository when you want to delete it from what I've read in the documentation. So the command should be:

ansible-galaxy delete 030 ansible-firewall


How can I add an additional IP / hostname to my Kubernetes certificate? I've recently followed this guide to set up Kubernetes on my local Raspberry Pi cluster. I can access it fine on my LAN; however I'd like to be able to access it remotely as well. I've already set up port forwarding on my router, but when I try to connect from outside the LAN, I get a x509 certificate error indicating that the certificate is signed for my local 192.168 address but not for my home's IP. I could bypass this with the --insecure-skip-tls-verify options but I'd much rather have the certificate used simply incorporate my IP address. Only... I'm not fully sure how to do that, though I do have some ideas.

When I initially set things up with kubeadm init, should I have used the --apiserver-advertise-address option to specify my home IP? Would doing so have had any adverse effects (such as precluding my local/192.168 IP?). Ideally I'd like the certificate to be signed for both the internal and external IPs. Or is there some other option I should have given to kubeadm init? Secondly, now that it has been initialized, is there any way to regenerate the certificate and swap it out, or will I need to nuke the cluster?

To clarify, I don't have any certificate managers installed. I'm not talking about Ingress. I'm talking about the actual "root" certificate (for lack of a better term) that kubeadm generates when first installing the cluster, i.e. the one that lives at /etc/kubernetes/pki/apiserver.crt

To do this, you’ll first need your kubeadm configuration file. This creates a file named kubeadm.yaml:

kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml

Now open the file in an editor, and find the certSANs list under the apiServer section. If it does not exist, you’ll need to add it; if so, you’ll just add another entry to that list. Example:

apiServer:
  certSANs:
  - "172.29.50.162"
  - "k8s.domain.com"
  - "other-k8s.domain.net"
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s

Now move the old certificates to another folder, otherwise kubeadm will not recreate new ones:

mv /etc/kubernetes/pki/apiserver.{crt,key} ~

Use kubeadm to generate new apiserver certificates:

kubeadm init phase certs apiserver --config kubeadm.yaml

Now restart your kubeapiserver container:

Run docker ps | grep kube-apiserver | grep -v pause to get the
container ID for the container running the Kubernetes API server
Run docker kill <containerID> to kill the container.
The Kubelet will automatically restart the container, which will pick up the new certificate.

If everything is working as expected, don't forget to update the kubeadm ConfigMap stored in the cluster, otherwise, future kubeadm upgrade will be lacking your new config:

If using Kubernetes < v1.15:

kubeadm config upload from-file --config kubeadm.yaml

For Kubernetes version >= v1.15:

kubeadm init phase upload-config kubeadm --config kubeadm.yaml

This article has a more complete guide on how to Adding a Name to the Kubernetes API Server Certificate


Create clean/blank MacOS fs on MacOS or Linux In developing/testing OSS libraries I can replicate a clean Linux fs using containers. However, if I want to test oss libs for MacOS end users, I seem to be in a bit of a quandary. Is there any way to create a clean fs on MacOS that I can test libraries with? Perhaps using chroot? Can someone explain exactly instead of just saying "use chroot"?

I don't think you can containerize MacOS, but you could create a Mac OS VM and snapshot it, that way you can easily restore to the same exact Mac OS version for each test.


DNS hosting choices - how to programmatically update DNS settings I manually update the DNS settings at the registrar for my domains, such as

https://domains.google.com/registrar

What are people using to automate updates to DNS settings? which registrars have an API ?

Specifically I need to manually update :

TXT field for the DKIM setting
TXT field for spf

@                  TXT          1m       "v=spf1 ip4:100.22.22.100 include:blah.com -all"
A record everytime I change external IP

To run my own DNS host servers seems like overkill. Ideally I would just make some secure REST call to perform programmatic updates.

Amazon web services provides such a service,  called Route53. You can actually use REST calls,  but it is much easier to use their command line tools or bindings for PHP,  Python or JavaScript.

Here is an entry point to get you started, the route53 change resource record sets call can be used to update or insert  one or many records on a domain.
 https://docs.aws.amazon.com/cli/latest/reference/route53/change-resource-record-sets.html

Of course you need to create an account, set up permissions and delegate name servers to route53.

It's also not free,  but fairly cheap at $0.50 for first 25 domains, then $0.10 after.  All charges per month.