Pros/cons of discontinuing a DevOps workflow? I'm trying to evaluate whether or not it is a good idea to move away from a devops-style workflow to the traditional dev-then-ops (not sure what you call that).

We are a small 5 person department tucked away within a 4000 employee traditional media (eg non-software) company.  Two years ago we began building software to allow our department to significantly scale up our production. We've been pretty successful and the greater company is beginning to take notice.  To date, we have been solely responsible for the design, development and deployment of what has become a ~10 service AWS microservice platform.  Our team doesn't identify as DevOps, but without question we are living the DevOps life, with each developer intimately familiar with both the code and the system it runs on.

One of the questions we will face shortly is what "efficiencies" are shared between us and the IT department for our parent company. Our project owner usually prefers outsourcing over in-house learning, so in our case these efficiencies probably means getting as much IT work "off our plate" as possible.  Currently, I would say our team has a 70/30% split between experience in coding vs infrastructure.  The IT department is solidly in the IT realm, with no visible crossover into software development.

Our project owner (a non-technical individual) is hoping that by handing off as much work as possible to the IT team we will see a ~1:1 boost in productivity for each hour of operations work that we shed. I'm skeptical about this though.  Our product is still pre-beta (despite already being a significant business asset) and in our limited experience with the IT department there are usually significant delays for things as simple as filesystem permission changes.

Right now, my ideal solution would be for the IT department to "adopt" us and allow us to continue deploying our own work, while ensuring that we meet the standards & requirements of the IT office.  I'm not sure how realistic that is though.  Plus it is nearly the opposite approach our project owner is advocating since it would add additional operations work in the short term.

In our situation, what are the likely pros/cons of staying with DevOps approach vs handing off IT?

It's not a good idea.

In my experience you'll gain the disadvantages of both while any projected advantages will somehow fail to materialize.

Itemized:

You will lose speed.
IT will comply with their own standard. The new task (for them) will follow the same 'sluggish' template all their work now has. Be prepared they will find it challenging - so even less speed than standard simple actions.
You will fail to offload.
IT will lean on you guys for every single anomaly. You will put in effort to get one guy up to speed -  and the next thing you now you'll be repeating yourself because the following task/problem/day there will be a new guy, again.
Documentation will be required, but it will not help.
Again template behaviour will be that short manuals will fail to catch every anomaly, and thorough texts will not be read for being too long. So any investment here will be a loss, likewise the enormous effort needed to implement improvements to get your tools to 'shrink-wrapped'quality.

Last but not least any problems will reflect on you guys. Tar, tarbrush principle.

If the above sounds cynical, well, I'm afraid I've been there. Repeatedly.

What to do instead?

Go shopping in the IT department, find yourself a useful candidate, and keep this guy 'on loan' to relieve you workload.


nodes in a kubernetes cluster in azure unable to communicate with one another I'll start off by saying I'm not really an infrastructure person, but am being required to maintain it until we hire someone more suited to it, so apologies if I have any terminology wrong.

We have a kubernetes cluster (AKS)in azure.  Currently we are running only 1 server in the cluster, with 4 namespaces configured within it, representing each environment.  To properly support our production environment, we ideally want at least 3 servers, with the ability to scale appropriately.  However, whenever we try to scale it up, services deployed to different server nodes cannot communicate with one another.

The unfortunate thing is I have no idea even how to start begin debugging this.  I know we have some custom nginx rules that microsoft gave us, but even setting allow everything as the first rule does not seem to resolve this issue.  How do I begin debugging this?  What information would even be helpful for this issue?

With help from a Microsoft Engineer, we were able to resolve our issue.

The trick to it was that the node that could communicate with the outside world was always the one that the kubernetes tunnel pod was running on.  This indicated a problem with communication between nodes.  The traceroute indicated that this, with no network security group traffic restrictions, was bouncing between the node1 ip, to the load balancer ip, back to the node2 ip.

we had a security rule that restricted all traffic that was not on our specifically approved list.

One of the debug steps I had taken was to open all traffic between the load balancer ip to the load balancer ip.  However, this failed to consider the fact that the traffic was not from the load balancer to the load balancer, but from nodex to the load balancer then from the load balancer to nodey.

We added an inbound and outbound traffic rule that allowed traffic from the subnet ips that nodes can possibly be on to the load balancer, as well as within the same subnet for good meansure.  This resolved the issue.

If that's confusing, I can provide images that show the new rules and explain it a little clearer, but I'm hoping that is clear enough for anyone who experiences a similar issue.


How to handle availability zones and failover with ansible? Is it even within the scope of ansible? I am only getting started with ansible.

I want to set up a new development infrastructure with the infrastructure as code principle. First thing I'll have to prove is a Jenkins instance with a few nodes for a Java dev team.

I want to do it correct right from the start, so I want to include some redundancy. Let's assume that in this scenario there are 3 sites with bare metal servers. I want to set up a Jenkins master in all 3 zones which form a cluster.

But I don't want to think about that myself. I just want to provide the info to ansbile what server is in which availability zone, and ansible should make sure to set up a Jenkins master in each availability zone and connecting it to the cluster.

Maybe this is already out-of-scope for ansible and I need another tool on top of ansible?

Any thoughts?

What your are describing can typically be addressed with a correct inventory design.

[zone1]
serverA
serverB
serverC

[zone2]
server1
server2
server3

[zone3]
serverX
serverY
serverZ

[jenkins_master]
serverA
server1
serverX

[jenkins_slave]
serverB
serverC
server2
server3
serverY
serverZ

From there you can easily:

create plays that will target a group or a pattern and that you can limit to an other group or pattern at runtime, e.g. target jenkins_master and run the playbook on all of them or limit it to zone1.
create inventory vars for your hosts and groups so that e.g. servers in zone1 will know what is the uri of the master in that zone.

This is a super-quick overlook. I suggest you read in depth working with inventory to get a better understanding of all the inventory concepts and pay attention to the part concerning the organization of variables in the inventory for groups and hosts


Scale number of pods in OpenShift from oc command line I am new to Openshift , and want know how to change the number of pods of  specific deployment from oc command line.

I tried to do:

oc scale dc name_of_deployment--replicas=2

Error from server (NotFound): deploymentconfigs.apps.openshift.io "name_of_deployment" not found

My deployments were installed with helm, if i do

oc  get dc

I am getting some other deployment but not the one that I want to scale.

Note: The deployment that I want to delete is running without any issue.

found the issue,  instead of 'dc' need to use 'deploy'

oc scale deploy name_of_deployment--replicas=2


ChaosMonkey for k8s? Is there a kind of tool like ChaosMonkey specific for k8s that kill nodes and pods during office hours to train the team how to deal with issues and to harden the k8s cluster?

It is possible to create a script or program that kills random PODs, but if such a tool already exists then this will save time.

You can take a look at kube-monkey. According to the README, it randomly causes failures in kubernetes clusters.


Docker compose scale, what is it good for? I have a docker-compose with 4 containers deployed in production.
I wanted to use docker scale for the nodejs part. (with nginx LB)
If the docker-compose scale is done on the host machine, how does it suppose to improve performance, I did not limit the resources for that container, it means that it can burst as much as the host allows it.
What docker-compose scale is good for anyway?
Thanks

Scaling is increasing the number of containers for any specific service. This is primarily helpful for a cluster, not on a single host setup like yours. When you scale your application up in a cluster, multiple containers of the same service get spread across the cluster (Based on their availability (Resource or anyway)). This makes the distribution of load across containers an available benefit. In a single host environment, with limited resources, scaling won't do you much good as the processes are going to be the slave of the same schedular, so you won't have that parallel servicing.
So now your question, why does scale in docker-compose exist? From the top of my head in no specific order of value:

You can minimize your services' downtime (going as low as none) while upgrading with docker-compose's scale functionality.
If an app can handle only so many inputs, in other words, if there's a queue of inputs, you can essentially increase the available queue space by scaling the container up and have multiple containers handle a portion of the inputs, reducing input loss.
The more obvious one, if one container goes down, there will still be at least one replica left.


How do I use only an encrypted version of my AWS RDS password in my terraform code? Is there a way to only have an encrypted version of my RDS password in my Terraform code and for AWS to decrypt it for RDS so you use the decrypted password to login to the RDS?

So set my rds password variable value to the encrypted password and when terraform builds the RDS I login in with is the unencrypted password?

If I am understanding your question correctly, what you are describing sounds like a perfect use case for SSM. You can store your RDS password as a secure string parameter encrypted by KMS, then reference it in you terraform file as a data source with decryption. With this method, you only have to reference your decrypted password, and never directly store the encrypted or plain text in your tf file.

data "aws_ssm_parameter" "foo" {
  name = "foo"
  with_decryption = true
}


How can I develop and support mainframe software without going broke? Mainframe (zOS) software is still very much in use right now. Many of the most critical applications of banks, insurance companies, airlines, global companies, etc are running on (good old) mainframes.

However, for companies that are in the business of developing and supporting mainframe softwares, it may be (terribly) expensive to have the required infrastructure for developing, testing and supporting such software.

What are possible solutions for developing and supporting mainframe software without paying fortunes for the required infrastructure (and spending all software revenue to it)?

One possibility would be to find a 3rd party business owning such infra and selling on-demand usage/access time/connectivity to it as a service for more convenient expenses. For example Mathru Technologies (found through plain google search).

Another possible approach is to obtain such services from strategic partners or maybe even your very customers. Maybe even for free, depending on the quality of your relationship.

You could also ask the mainframe community for advice and/or leads in this direction.


How to make the Jenkins agent build then start the server? I want the Jenkins agent to build and start the server.

I do this by specifying how to build and start the server in the Build step of the project configuration in Bash

echo "Installing dependencies"
npm install
echo "Starting server"
./startscript.sh 3000 & #this script will start the server with specified port

Why the ampsersand in /startscript.sh 3000 &?

I noticed that without the ampersand & in the last command, the server is started as expected (I can access it from browser). However, the Jenkins build hangs and cannot finish (because the script starts the server and won't return until the server is terminated).

So I tried to fix this with adding the ampersand to make the script process runs in background. With this, Jenkins job successfully ends the build.

But adding the ampersand still doesn't fix
The server process is still terminated (it actually started because I checked the log and saw that it was started). I think that this might be because when Jenkins finishes building, it exits the shell and sends the SIGHUP to attached background jobs.

Adding nohub still doesn't fix

Running the last command with nohup ./startscript.sh 3000 & still doesn't fix the bug.

Question

How to make the agent build then start the server.

Note

The console output of Jenkins builds shows both the command and the output of it (like the behaviour of the -x flag of the shell). And the last command was shown like this:

+ nohup ./startscript.sh 3000
Finished: SUCCESS

It doesn't show the ampersand &... So I don't know if the ampersand was actually correctly interpreted during the run.

You are running into a conflict between jenkins nature of making sure everything is done before a job is complete and your desire for some long-running things to be asynchronous.  There is no right answer here, both are reasonable desires, but you and jenkins are pulling in different directions.  It may be ok for some parts of the jenkins job to not be completely done but they are good enough for the job.  This is a decision you get to make.

If getting this working inside of jenkins is too much trouble there are work arounds.  You can make your own daemon that looks for job completions (via files or database entries) and then starts something else.  Obviously this means you lose the UX features of jenkins that let everybody know where something is.  But you could make this work inside of jenkins by having different jobs also.  The quick job finishes and then the longer job takes over.  Since each is its own job it can have timeout requirements that make sense for it.


Using the built binary of a docker image in another docker image Problem

I have two repositories A and B. I have a docker image registry, where I can push the docker images made of the repositories A and B.

I need only the binary of A's source code to be included in the docker image of B. So when running the docker image of B, the jar file of A is present and can be used. And in case of updates in repository A, the registry image is subsequently updated.

Question

I've searched now several threads and the docker documentation. Is there a recommended way to make this happen with docker respectivly dockerfile commands?

So far I tried the following and it worked, but it wasn't so intiutive. Is this the recommended way?

FROM myregistry/my-A-img AS A
FROM ubuntu-base-img

# Build commands for B ...
COPY build-B.jar /app

# Get the A image from the registry
COPY --from=A /path/to/build-A.jar /app/lib

# Run B binaries

What you need is to use the build pattern or the multistage features of docker

https://docs.docker.com/develop/develop-images/multistage-build/

With that, you can use as many images as you need depending on your build process


Is there an image on dockerhub that is able to run make? When one searches for make on dockerhub multiple images are returned, but none of them seems to be able to run make.

GNU make can be found in the gcc container.

  Compile your app inside the Docker container
  There may be occasions where it is not appropriate to run your app inside a container. To compile, but not run your app inside the Docker instance, you can write something like:

  $ docker run --rm -v "$PWD":/usr/src/myapp -w /usr/src/myapp gcc:4.9 gcc -o myapp myapp.c

  This will add your current directory, as a volume, to the container, set the working directory to the volume, and run the command gcc -o myapp myapp.c. This tells gcc to compile the code in myapp.c and output the executable to myapp. Alternatively, if you have a Makefile, you can instead run the make command inside your container


How to publish a docker image to a private docker registry using Jenkins' blueocean? It seems that the syntax that is used in the pipeline plugin is incompatible with Blueocean. It looks like that the syntax is different as blueocean uses a declarative pipeline.

Example

http://fishi.devtail.io/weblog/2016/11/20/docker-build-pipeline-as-code-jenkins/

  stage ('Docker Build') {
    // prepare docker build context
    sh "cp target/project.war ./tmp-docker-build-context"

    // Build and push image with Jenkins' docker-plugin
    withDockerServer([uri: "tcp://<my-docker-socket>"]) {
      withDockerRegistry([credentialsId: 'docker-registry-credentials', url: "https://<my-docker-registry>/"]) {
        // we give the image the same version as the .war package
        def image = docker.build("<myDockerRegistry>/<myDockerProjectRepo>:${branchVersion}", "--build-arg PACKAGE_VERSION=${branchVersion} ./tmp-docker-build-context")
        image.push()
      }
    }
}

results in:

WorkflowScript: 5: Unknown stage section "sh". Starting with version 0.5, steps in a stage must be in a steps block. @ line 5, column 1.
   stage ('Docker Build') {
   ^

Attempt to solve the issue

When a steps block was added the pipeline failed again:

WorkflowScript: 13: Method calls on objects not allowed outside "script" blocks. @ line 13, column 13.
               docker.withRegistry

You can try to use scripting syntax into the declarative pipeline. For some step there is no declarative syntax yet. I had the same problem trying to use the docker global variable as a step.

stage ('Docker Build') {
  steps {
    // prepare docker build context
    sh "cp target/project.war ./tmp-docker-build-context"

    // Build and push image with Jenkins' docker-plugin
    script {
      withDockerServer([uri: "tcp://<my-docker-socket>"]) {
        withDockerRegistry([credentialsId: 'docker-registry-credentials', url: "https://<my-docker-registry>/"]) {
            // we give the image the same version as the .war package
            def image = docker.build("<myDockerRegistry>/<myDockerProjectRepo>:${branchVersion}", "--build-arg PACKAGE_VERSION=${branchVersion} ./tmp-docker-build-context")
            image.push()
        }
      }
    }
  }
}


Is Continuous Delivery the Control Framework for DevOps? I've been reading Jim Bird's DevOpsSec book, one of the statements in Chapter 4 - Security as Code is as follows:

  Agile ideas and principles—working software over documentation, frequent delivery, face-to-face collaboration, and a focus on technical excellence and automation—form the foundation of DevOps. And Continuous Delivery, which is the control framework for DevOps, is also built on top of a fundamental Agile development practice: Continuous Integration. (O'Reilly 2016, ISBN 9781491971413)

My understanding of a control framework was that it declares certain Control Objectives such as:

Payments are made only for authorised products and services received.

This may be satisfied by some process built into the system, and would likely have a number of checks and balances; however, you wouldn't say that that process is the Control Framework.

To me it feels that that sentence should read: "And Continuous Delivery satisfies the Control Framework for DevOps".

Am I totally wrong? Is Continuous Delivery the Control Framework for DevOps?

Or, am I right and Continuous Delivery is not the control Framework for DevOps?

In my opinion you are right, Continuous Delivery (CD) is not the control framework of Devops, at least it is not the only possible one.

But in the context of the book you're quoting it gets the most used possibility when you start to include security baseline and assessments as part of the product delivered.

In a security context, you'll add smoke tests in a post deploy phase to ensure your application is not subject to XSS or Sql injection for example. If any of this tests fail, block any deploy in other environment, mark the deploy as failed and eventually rollback to previous version.
This enforce the security rules to be satisfied, and in this way act as a control framework to ensure production deploy are 'safe' for known vulnerabilities.


Running a command as a specific user on an ec2 using ssm Background

I want to automatically configure an EC2 via a build, using the aws cli and ssm.

The (manual) setup for the container looks something like this:

Create an EC2
Run a shell script as root
Run a shell script as a specific user

Question

Is it possible to run a command on an ec2 utilising a tool such as aws ssm send-command specifying the linux user which will execute the command?

Assuming the command is a shell script, would specifying the user inside the script do the same job? e.g using sudo su my_user

I didn't find anything on AWS or boto3 docs that allows for that, but I was able to execute as a different user using the runuser command. In theory, you could do the same thing with a combination of sudo and su commands, but this one is pretty simpler.
For that, you can do as follows:
runuser -l  userNameHere -c '/path/to/command arg1 arg2'

Since send-command executes as root, you don't have any issues.
Note: I thought that send-command uses in some way a session managed by the SSM Session Manager, but I was wrong. I spent a good time configuring SSM Session Manager preferences and tagging IAM resources according to this doc and this one, but send-command always execute as root as far I saw.
Sources:

https://man7.org/linux/man-pages/man1/runuser.1.html
https://www.cyberciti.biz/open-source/command-line-hacks/linux-run-command-as-different-user/


Apache Airflow federation - observe remote dag in full detail? We have numerous airflow instances running in different organizations/teams around the company.  While we have built a highly scaleable centralized airflow service, we do have cases where we need to depend on another airflow instance to run a dag.

Our DevOps team often gets tasked with running larger orchestrations across teams, and we prefer to keep a highly detailed view of all tasks, even if they are in a remote airflow.

Is there any good way to monitor, with full fidelity, all of the tasks in a remote airflow?  Something similar to the sub-dag operator for a remote airflow would be sufficient.

I spent a long time on this problem and made 2 custom solutions as nothing out of the box exists.

Both steps require you to either use both the experimental and the extra plugin API or to have access to the remote database.

At that point, basically either:

Make a GUI plugin on a separate page that hits the APIs and shows all tasks (list exec dates, list tasks in exec date, get status of each).
Make python operator steps in your dag to call out to the state of the remote dag steps with the same name.  It's hard to reflect anything outside of running/done though as you need your step to keep checking in case the remote end restarts the step/tries again.

The APIs are slow, so this isn't fun.  I'd recommend going for the DB if you have the option.

Airflow 2.0 will fix the APIs apparently, and then this may be a lot easier.


Autoscaling with Kubernetes daemonset I have a Kubernetes cluster running on Google Compute Engine. My program uses stateful connections, so I do not want my pods being moved between nodes. Because of this, I am running my workload as daemonsets, and giving each node the amount of compute power that one pod needs.

This system is working, but I would like a way to autoscale. Normally, the high workload would produce more pods, which would then trigger the creation of more nodes. But since I am only using daemon sets, this normal solution will not work.

I can use commands to increase or decrease the number of nodes in the server, but I would like this to happen automatically. Ideally, when the workload is high, the cluster would spawn another node in its node pool. Even better, if there is a solution to autoscale down the number of nodes, while minimizing closed connections.

I haven't found much information on ways to do this. I am not very experienced with dev ops, kubernetes, google cloud, docker, etc. so I would appreciate explanatory answers.

Thank you.

I think you can do this by Nodeselector. Firstly add a label for node selector in your daemonset config. Then label your nodes with the attached label. Now if you can set the autoscaling thresholds, it will be deployed on that node automatically on nodes that matches the label. Maybe you can tweak somehow to attach a label to your node when it is added to your pool.

Also, with Kubernetes 1.6+, we have the feature of rolling update strategy which is far better than onDelete.

Cheers


Docker exec environment different to container shell I have a docker image that looks something like this:

FROM ubuntu:16.04

# Install packages for building ruby
RUN apt-get update && \
apt-get -y install build-essential git curl wget net-tools apt-utils sudo man \
libssl-dev libreadline-dev libyaml-dev nano gcc rubygems ruby-dev

# Install rbenv and ruby-build
ENV PATH /root/.rbenv/bin:$PATH
RUN git clone https://github.com/sstephenson/rbenv.git /root/.rbenv && \
git clone https://github.com/sstephenson/ruby-build.git /root/.rbenv/plugins/ruby-build && \
/root/.rbenv/plugins/ruby-build/install.sh && \
echo 'eval "$(rbenv init -)"' >> /etc/profile
echo 'eval "$(rbenv init -)"' >> ~/.profile && \
echo 'eval "$(rbenv init -)"' >> ~/.bashrc

# Install ruby and bundler
# Install Bundler
ENV CONFIGURE_OPTS --disable-install-doc
RUN rbenv install 2.4.1 && \
echo 'gem: --no-rdoc --no-ri' >> ~/.gemrc && \
rbenv global 2.4.1 && \
rbenv local 2.4.1 && \
gem install bundler

ENTRYPOINT /bin/bash

My problem is that when I build the image and start a container

docker build -t test .
docker run -it --name test-container test

If I run the commands directly in the container shell,
ruby is 2.4.1 (as I'd expect after running rbenv global 2.4.1)

ruby -v    # 2.4.1p111
which ruby # /root/.rbenv/shims/ruby

But if I open a new shell and use docker exec,
ruby is 2.3.1, which is the system default

docker exec test-container ruby -v    # 2.3.1p112
docker exec test-container which ruby # /usr/bin/ruby

I think it could have something to do with the bash profile not starting rbenv properly when using docker exec. The env command seems to have less output when using docker exec too, which is why I suspect something to do with .bashrc, .profile, or even /etc/profile not loading somehow.

How can I ensure docker exec produces the same result as running the command directly on the container?

Please refer to bash initialization process.
https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html

In your Dockerfile, at lines 12-14 you are patching files, but do not activate any of rbenvs.
Later at your last RUN call, you finally ARE activating an rbenv. This starts pointing at that specific version of ruby (I thin this patches root files)
At that point you have different global shell initialization files vs. user initialization files.

Difference

entry point: "non-login" initialization process happens, then the file: /root/.bashrc is executed (a.k.a. "sourced"), it points to your cool ruby
exec of ruby: ruby executable is located using your default PATH. without running /root/.bash* files.

To validate all this you can append printouts to your shell's initialization files, and see what gets printed upon 2 cases.

Regards.


How to measure human utilisation? In The Phoenix Project the single graph in the entire book shows that as a person's workload increases into the high 90%s the wait time on them increases exponentially. In fact in the book it claims that:

Wait Time = (Percentage Busy/Percentage Free)

So if a resource is busy for 35 out of 40 hours per week then:

Wait Time = 0.875/0.125  = 8

However if a resource is busy for 39 out of 40 hours per week then:

Wait Time = 0.975/0.025  = 39

This is multiplied up by the number of waits in the workflow. It's obviously something we want to keep an eye on!

So, given all of this it's obviously vital that we keep people's utilisation at a sensible level. Given the book's insistence of the importance of this formula it offers little practical advice on how to measure these values. My question is, how can you practically measure a person's percentage busy?

TL;DR: It is the wrong thing to measure. By measuring and increasing utilization in employees across the board, you create problems in the system and reduce overall throughput.

Throughput Accounting

What you actually want to measure is throughput, inventory and operating expense all together and try to reduce inventory and reduce operating expenses while at the same time maximizing throughput. This method is known as throughput accounting.

In software development, inventory is the work in progress that is not bringing benefits to the customer yet. Anything that has been done, but not released. Throughput is the amount of work useful to the customer that is being released. Any work done that is not directly useful to the customer is accounted as operating expense.

Simple system

In a simple system with single human or multiple humans working independently with independent equipment as much as each one does will directly increase the throughput of the whole system. This leads to the common misconception that is basis for this question that increasing human utilization will lead to increased throughput in all systems. But you still measure the throughput of the system, inventory and operating expenses.

Complex System

In a complex system, even with as little as two dependencies, increased utilization of one part of the system can directly lead to decrease of utilization in the bottleneck, which decreases throughput of the whole system. Any increase in productivity outside of the bottleneck is a mirage.

Example: Team of software engineers has all code reviewed by the software architect, who also makes plans for new features. This person is a bottleneck, code not reviewed by the architect will simply increase inventory, if the architect does not have time, no new features will be properly planed. If you start measuring utilization of software engineers, they will each try to make more changes, rather than better changes. The time the architect will need to spend on each change will increase and the total time spent reviewing will increase further by the increased amount of changes to a point where no time would be left to plan new changes. Eventually the whole system grinds to a halt. If on the other hand they decrease utilization, even spend time idling, they spend longer on each change or peer reviews, this could lead to decrease in time required for reviews and eventually increase in throughput. This is just single team with 2 dependencies. Engineers depend on architect for new changes to be planed and for changes to be reviewed.

Clearly the benefits are to be gained in properly managing the bottleneck and trying to gain productivity at the bottleneck, where hour gained, is hour of throughput of the entire system.

This is the real message of The Phoenix Project and comes directly from the Theory of Constraints by Eliyahu M. Goldratt. You might also read an article on utilization thinking vs throughput thinking. I would also suggest to read more on Critical Chain Process Management.

Remember: What you measure is what you get. And you definitely DO NOT WANT to get increased individual utilization. A road to Hell is paved with good intentions.


how is software defined infrastructure related to devops? I'm a new developer who is picking up python. I'm currently writing code using python's asyncio or twisted framework to automate the setup of some raspberry pi and servers I got 2nd hand (xen, hyper-v) using SSH, I tried ansible but I prefer writing my own code for learning.

does this fit into software defined infrastructure?
if I setup automate deployment of code, is this considered devops?

additionally, I hope to enquire if my intended workflow is considered "devops"?

gitlabs
automated code testing (tox/travis ci)
automated deployment of code to raspberry pi

if I setup automate deployment of code, is this considered devops?

Sure, automating everything is a big part of DevOps.

  I tried ansible but I prefer writing my own code for learning

Now, this is laudable, but using Ansible (or Puppet or Chef or ...) would be "learning" as well. The benefit of using established tools over DIY is that if/when you meet other developers, you have a common toolset to work with. Doing it yourself is nice sometimes as well, but basing a whole software development process on 100% self-written tools (from scratch) is not a good idea, unless you have deeply evaluated the existing tools and consider them all unfit for the purpose.

When your team grows and you encounter a new developer - is it more likely that they already know about Ansible, or that they know about the code you wrote?

Also, the fact that the tools mentioned above do not do their stuff in the "code" directly, but that they have text files which configure them, is important as well. It abstracts the configuration stuff away from actual code, which will be hard to maintain a few years down the road, no matter how good you are.


When to "npm publish" when using git flow I'm using the following resources as a guide to git flow:

git-flow cheatsheet
git-flow
Using git-flow to automate your git branching workflow

However, none of them explicitly mention at what point you perform the npm publish from. Do you do this after you have bumped the version number but before you finish the release? This would mean the publish is coming from the release branch. Or do you finish the release then perform the publish from the master branch?

Thanks.

I'd set up my CI system to do this on every commit to master, that is, after the release branch merges.

The first very straightforward reason for this is that the commit to master gets tagged with the release, and you want the thing you ultimately publish to match exactly the version of the source code at the tag.  While there shouldn't be any merge surprises if git flow is run perfectly, it's still better to publish the content of a tag and not the commit immediately before a merge.

The second reason is that it is a very simple rule that can be automated.  If you do any amount of final testing on the release branch, you need to make a manual decision as to whether or not you can release or not; your CI system (probably) can't make that decision just based on the presence of a release branch.  On the other hand, since every commit to master is merging in a release (or hotfix) branch that you've fully validated, the commit itself is a signal that you're ready to release.


DMAIC - Define, Measure, Analyse, Improve, Control in the DevOps world? Six Sigma's DMAIC (see also Wikipedia) has been there before continuous procedures, so I think more or less these concepts have been nourishing itil, then CI/CD and now DevOps as well.

Still I am not sure whether we have an established data model for example to allow DMAIC across many iterations.

Question: do you know of examples of practices/ aligning DMAIC with DevOps (at the measure side, first of all)? Does it make sense?

Yes, DMAIC makes sense. If processes like CD need to be improved it is important to have a baseline. Therefore the Measure step is useful. The team investigates the status of process X, e.g. CD by documenting the current status.


Provisioning a server on behalf of a user on GCP/AWS/Azure and probably relinquish access I'd like to build a service where I run a custom setup script on behalf of a user on an t2.micro equivalent instance but be able to probably relinquish access to the server. Any advice on the tech stack and flows that I could use to accomplish this?

Let's assume I have unlimited access to AWS resources to create as many instances as I like. The service is targeted at users unfamiliar with how even ssh works and we are basically setting up a custom software service automatically that runs on their dedicated service. But we need to prove that there is no way we can gain access to this user's server again.

Any help about creative ways to accomplish this?

Rather than provisioning resources on-behalf of a user which would require them exposing their credentials to you in some way, I would implement this using the various marketplaces:

Azure: Virtual Machine Offer Publishing
AWS: How to List Your Product in AWS
GCP: Sell on GCP

This has a couple of advantages:

It's fully automated, you are never exposed to their subscriptions thus you don't need to prove that you no longer have access - you never had it in the first place.
Everything is created within the customer's Subscription (Azure), Account (AWS) or Project (GCP) thus they bear all of the costs, you then get money from the cloud provider based upon how you price yourself in the marketplace.
You can generally create more than just a VM, i.e. you could create a PaaS Database for persistence.


On Azure, how do I run a short-lived Docker container on a schedule? I have a fairly simple Unix shell script packaged up in an Alpine Linux Docker container hosted on an Azure container registry. A VM runs this script with cron:

docker login <snip>
docker pull example.com/bar:latest
docker run  example.com/bar:latest

Can I do without the VM and use Azure services instead, perhaps with some sort of scheduler running this on an Azure Container Instance?

My motivation is not wanting to maintain and pay for the VM.

Azure Container Instances
 (ACI) may be a good option as you suggest.  These let you run a container directly on Azure, without having to manage a VM, with per-second billing for the time the container is used.

Although one of the demos on that blog mentions Kubernetes, the idea of ACI is that you can create a container through the Azure CLI with az container create, just like on your local workstation with docker create.

To create the container, you can use Azure CLI (az command, see quick start docs) or Azure Cloud Shell.

You would need to create/run the container on a schedule from somewhere else - Azure Functions might be a good place to run the "container create" command from  a scheduled function.  This supports bash, PowerShell, and other languages - all running on Windows.

If you want to keep using Docker containers without running VMs or learning Kubernetes, this might be a good option.

Alternatively, you could move all your code into Azure Functions, but that's a bigger decision.

Update: Jan 2019 - Azure Logic Apps can be used to run scheduled tasks as well.


In what way does a new ECS Fargate service not satisfy the task definition's required capabilities? When creating a new AWS ECS service using a Fargate task, in the AWS Console wizard, it fails with the following error:

Failed creation of Service
Service creation failed: The specified platform does not satisfy the task definition’s required capabilities.

The service launch type is set to FARGATE and the Platform version is set to LATEST. The task is attempting to mount an AWS EFS filesystem for persistent storage (introduced in version 1.4.0 of the platform).

This is by design because AWS have yet to upgrade LATEST to point to v1.4.0 (as per 2020-06-23, see AWS ECS Developer Guide). Set the service platform version explicitly to 1.4.0 (instead of LATEST).


Jenkins Docker Behind Reverse Proxy Adds ":80" to the URL The Jenkins container redirects to URL:80, which is causing an error. When I remove the :80, everything is working as it should.

I have tried the proxy with a simple web app and there were no issues.

This is the docker-compose.yml https://pastebin.com/djJTuBZL.

The cert is correctly issued and the connection is secured. The host I'm using is live and correctly configured with ports 80 and 443 open.

The container exposes 8080 and this is the same as the VIRTUAL_PORT.

Also, I would have assumed that accessing an address with :80 wouldn't have caused any issues this being the standard HTTP port eg www.google.com:80; unless I'm missing something.

If this helps this is the logs when I run docker-compose up https://pastebin.com/NUbDL7PC.
Here are the logs when I try to do the first login, the space is before I remove the :80 then load the page https://pastebin.com/0wxGDBzi.

How can I resolve this problem?

If it's a brand new Jenkins, you can try creating an SSH tunnel directly into the container with port 8080 and checking if the problem also appears there. That will at least allow you to check if it's Jenkins redirecting you or Nginx.

There you can also try adjusting "Jenkins URL" in "Manage Jenkins" -> "Configure System".

Alternatively, check  jenkins_home/jenkins.model.JenkinsLocationConfiguration.xml. jenkinsUrl tag is also present there.


Can use a private IP address as the puppet certname? Im very new to puppet but know in /etc/puppetlabs/puppet/puppet.conf I can set cert name to a url form following linux accadamy vidios but can you also set it to a private ip address.
Im trying to set up a puppet server and agent on my personal aws account replicateing what i did on my linux accadamy lab servers on ec2s except im trying to get it working with amozon ec2s rather than ubuntu 18.04.

I don't have a url set up for my ec2s as that costs extra can I use the ec2s private ip addresses rather than a url?

You can indeed use IP address in the "certname" directive of puppet.conf file.

In addition, you can create a private hosted zone in AWS Route53 that is called for example lab.local and then create an A record for your puppet server such as for example puppetmaster.lab.local and then use this address in the certname directive.

Opening a hosted zone in Route 53 doesn't cost much... around 0.50$ per month for your first 25 zones.

The full Route53 pricing can be found here.


Best practice when automating Drupal (or other soft) installation regarding database settings Suppose I have Drupal and CiviCRM installations to automate.

These, however, need particular configs for working together (some file edits and database configs)

Should I make a fixed version install (eg. Drupal 4.7.5 + CiviCRM 4.5.6), install them and configure, and then save the edited files plus database?

Or should I put more effort in my build script and make some queries and seds to edit those configs in the database and files?

Your question sounds like you should have a look at what I call Features Configuration Management, aka FCM. FCM is not only about using Drupal's Features module, and not about Configuration Management (as introduced in Drupal version 8). Instead, it is a special case of Software Configuration Management, aka SCM, applied to Drupal.

Mostly because Features can be considered as a code generator, whereas that generated code may be considered as the build script (as in your question). Which can then also be used to be migrated through multiple environments.

Have a look at my (popular) answer to "How to use the Features module in a 3 dev environments?" for (way) more details about all this.


Single Build with Multiple Artifacts I'm a little confused on the Azure Pipeline steps.  My solutions has many projects and one UI project that contains both an ASP.Net-Core WebApi/MVC with Angular.

the Build Solution builds similar to the

steps:
- task: VSBuild@1
  displayName: 'Build solution'
  inputs:
    solution: '$(Parameters.solution)'
    msbuildArgs: '/p:DeployOnBuild=true
      /p:WebPublishMethod=Package
      /p:PackageAsSingleFile=true
      /p:SkipInvalidConfigurations=true
      /p:PackageLocation="$(build.artifactstagingdirectory)\\"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'

The default Publish Artifact is:

steps:
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: drop'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)'
  condition: succeededOrFailed()

I'd like to do something with 2 artifacts (this doesn't work):

steps:
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: webapi'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)\\MVCProject\bin\$(BuildConfiguration)\netcorapp2.2'
    ArtifactName: webapi
  condition: succeededOrFailed()

steps:
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: angular'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)\\MVCProject\clientapp\dist'
    ArtifactName: angular
  condition: succeededOrFailed()

What am I missing / What the right/better way to do this.
(moving angular out of the project does not solve this problem, and we aren't ready for that yet)

Here is my YAML for multiple artifacts from a single build.

pool:
  name: Hosted Windows 2019 with VS2019
  demands:
  - msbuild
  - visualstudio
  - vstest
  - npm

steps:
- task: NuGetToolInstaller@0
  displayName: 'Use NuGet 4.4.1'
  inputs:
    versionSpec: 4.4.1

- task: NuGetCommand@2
  displayName: 'NuGet restore'
  inputs:
    restoreSolution: '$(Parameters.solution)'

- task: VSBuild@1
  displayName: 'Build solution WebApi'
  inputs:
    solution: '$(Parameters.solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    msbuildArchitecture: x64

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: WebApi'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)'
    ArtifactName: 'dotnet-webapi'
  condition: succeededOrFailed()

- task: DeleteFiles@1
  displayName: 'Delete build.artifactstagingdirectory for Angular'
  inputs:
    SourceFolder: '$(build.artifactstagingdirectory)'
    Contents: '**'

- task: Npm@1
  displayName: 'npm install'
  inputs:
    workingDir: project/ClientApp
    verbose: false

- task: Npm@1
  displayName: 'npm run prod-build'
  inputs:
    command: custom
    workingDir: project/ClientApp
    verbose: false
    customCommand: 'run prod-build'

- task: Npm@1
  displayName: 'npm run dev-build'
  inputs:
    command: custom
    workingDir: project/ClientApp
    verbose: false
    customCommand: 'run prod-dev'
  enabled: false

- task: CopyFiles@2
  displayName: 'Copy Angular/Dist to build.artifactstagingdirectory'
  inputs:
    SourceFolder: project/ClientApp/dist/ClientApp
    Contents: |
     **
     !config.json
    TargetFolder: '$(build.artifactstagingdirectory)'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: Angular'
  inputs:
    ArtifactName: angular


How to test database applications in QA testing environments? Process

Our current process is:

Build process produces a windows setup (msi).

This setup consists of a web site deployment and also database scripts - which can create a new database or upgrade an existing version.
The QA person needs to snapshot their Application server and their Database server.
The QA person then runs the setup on their own windows server.
The setup installs a web site and updates the database.

The purpose of the snapshots is to give them something to rollback to - to compare any behavioural changes between different versions of the application.

Current Technology and Environments

We rollout to Windows Server 2008r2 and above.  The Database servers are Windows Server 2008r2 and above with SQL Server 2008r2.

Our QA is a bit mixed in terms of what Virtualisation is being used. Our teams work in two ways, one with SQL Installed on the server and another process with the traditional app server / sql servers being separate. The servers are running on mix of storage using SSD / SAN and SAS drives, each server generally has 60gb HD, running 4gb ram. SQL Servers have 8gb or more

Drawbacks

The biggest drawback is the speed at which the environments are backed up - in terms of speed and the amount of space they take up.

My Thoughts

One of my ideas is to either create an app / script or use some tool to backup / snapshot the application server and MS SQL database at the same time.

Questions

I would like to know how other people are rolling out changes with a similar process to what we go through.

Additionally:

What other tools are available for example?
Also what recommendations would you make?

The biggest problem with database backups is that you cannot simply snapshot them. You first have to make sure the DB is not mid-transaction when taking a snapshot, otherwise you will restore to an inconsistent state. In the case of windows, you can leverage the Microsoft SQL VSS writer to snapshot your database. You will want to be sure that when you take a snapshot of the VM, HyperV is capable of leveraging this VSS writer.

If you have a particularly large dataset, it may be better to leverage your storage filer to use delta-clones if your filer has this feature and it can help to address the size/time issue you are experiencing with backups.

In terms of deploying the new application to QA there are several ways to do this. One is to have an automated build system such as Jenkins, Travis CI or a similar tool automatically trigger a backup and deployment upon a code commit. This can be done by leveraging a configuration management tool such as SaltStack, Puppet or Chef (which all run on Windows) that can automatically set up and configure a SQL server for you and deploy your application if you are able to script a SQL install with PowerShell. Personally, I would recommend SaltStack as it's Reactor will allow you to broadcast a beacon when your backup/snapshot completes and trigger a build (Puppet lacks this capability).

One option for spooling up a VM each time is to leverage a system like The Foreman which has a HyperV plugin. You could then use SaltStack (vis-à-vis Reactor. This is probably the best way if you backup system blocks until the entire backup is complete, which I doubt) or your automated build system (Jenkins/Travis) interact with The Foreman's REST API to request a host be spun up. Once this comes up, it would automatically check in with your configuration management system to install SQL server and restore the snapshot in the new VM. If you combined this with an automated testing battery, you might even be able to implement a "Push on Green" system to repeat this process into your next environment if all the QA tests pass.

The advantage to using a piece of middleware like The Foreman is that it will allow you to deploy into any virtualization system or even the cloud and prevent you from being locked into HyperV and make your environment more portable.

I am sure there are other ways to get there, but this is one way I have used in the past to implement similar deployment workflows.


Deploying a single-page web application with a java backend on kubernetes and haproxy My team (full stack) team develops single-page web applications using react as a client and spring-boot java apps as back-end.
We deploy the applications on kubernetes. However, we have troubles making rolling updates of the kubernetes pods seamless. Here's our packaging / deployment process:

The CI pipeline builds a docker images packaging the java application as an executable jar file and the compile javascript/css assets that form the react client.
That image has 2 entry points: one for starting the jar file (java back-end) and one which starts an nginx that serves the client assets (js/css).
A haproxy cluster routes requests based on URL paths to either nginx or the java back-end.

This approach works fine so far.
However, during a rolling update, there will be a number of pods with an "old" java back-end. This leads to the situation that clients with a newly deployed version of the react client
make requests to old java back-ends, because haproxy simply routes api traffic to any available java backend.
As i understand haproxy seems to have support for sticky-backends based on some rules, but this defeats the purpose of having a cluster of backends for load-balancing.
How do others tackle this problem?

Modern way to achieve seamless roll-out in this scenario is not on the load balancing (i.e. haproxy) side, but instead via proper deployment planning, backward compatibility and/or feature flags.
The issue with load-balancer tinkering is that in the end of day even if you get sticky sessions right it's not going to solve your problem. The reason for that - imagine a user that opened your app in a browser with old react. Underneath, you deploy new java back-end. If the user doesn't reload her browser - she's still going to use old react app with new back-end no matter what (just because javascript is already loaded to the browser).
Instead, look at following example how you can achieve it using API versioning:

Let's assume that data model on one of your api end-points is going to change
Let's say your old api was /v1 (or even un-versioned)

Here is how you can solve it:

You create /v2 version of your api, register /v2 version in your react app
Your latest java back-end needs to still have both /v1 and /v2
You first deploy all java back-ends to your back-end deployment with /v2 support (all your UI is still /v1 - but this is fine, because latest back-end supports it)
Once all back-ends are deployed, you can start roll-out of your UI deployment with latest image (again, experience is seamless since all back-ends support both /v1 and /v2)

As you can see, in the above approach, version conflict is avoided and experience is seamless.
P.s. You might want to consider splitting your docker image in 2 - it's a little odd to have a multi-purpose image with 2 entrypoints. There may be little justification for this design.


GitLab pipeline stop at the stage of pulling an image from Docker hub Description:
I have created a test project to serve a simple HTML page with httpd:alpine image. The following are the parts of the repository:
index.html:

    <meta charset="UTF-8">
    <title>Rostami test page</title>

Hey guys!

Dockerfile:
FROM httpd:alpine

COPY ./ /usr/local/apache2/htdocs/

docker-compose.yml:
version '2.2'

services:
  myhtml:
  image: html_mrt:latest
  ports:
  - "9999:80"
  restart: on-failure
  container_name: HTML_Rostami

.gitlab-ci.yml:
stages:
  - build
  - deploy

build:
  stage: build
  script:
  - docker build -t html_mrt .
  tags:
  - docker

deploy:
  stage: deploy
  script:
  - docker-compose up -d
  - docker rmi $(docker image -q filter "dangling=true")
  tags:
    - docker

Issue:
In addition, when I execute gitlab-runner status on the GitLab-Runner container, it says:
Runtime platform                                    arch=amd64 os=linux pid=79 revision=a998cacd version=13.2.2
gitlab-runner: Service is not running.

I've tried to start the service by gitlab-runner start but nothing has affected.

Problem:
It's the log of the build stage:
Running with gitlab-runner 13.2.2 (a998cacd)
  on Test K-1ZsHyF
Preparing the "docker" executor
00:11

Using Docker executor with image docker ...
Pulling docker image docker ...
Using docker image sha256:81f5749c9058a7284e6acd8e126f2b882765a17b9ead14422b51cde1a110b85c for docker ...
Preparing environment
00:02

Running on runner-k-1zshyf-project-1-concurrent-0 via 806fd175c866...
Getting source from Git repository
00:03

Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/m.rostami/test/.git/
Checking out 82d339a9 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:04

$ docker build -t html_mrt .
time="2020-08-18T06:36:52Z" level=error msg="Can't add file /builds/m.rostami/test/docker-compose.yml to tar: io: read/write on closed pipe"
time="2020-08-18T06:36:52Z" level=error msg="Can't close tar writer: io: read/write on closed pipe"
error during connect: Post http://docker:2375/v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=8jgar93piqxu2s7y5yp9w2wtg&shmsize=0&t=html_mrt&target=&ulimits=null&version=1: dial tcp: lookup docker on 192.168.1.1:53: no such host
ERROR: Job failed: exit code 1

192.168.1.1 is the default gateway and nameserver of the host that the Docker daemon is running.
How to fix the problem?
Thanks in advance.

I image you have used the dockerExecutor in gitlab-runner. This means you try to use docker-in-docker.
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html
I suggest to configure the docker socket binding which might be less secure but very easy to handle.
To register a runner with docker-in-docker support mount the /var/run/docker.sock:/var/run/docker.sock
e.g.
sudo gitlab-runner register -n \
  --url https://gitlab.com/ \
  --registration-token REGISTRATION_TOKEN \
  --executor docker \
  --description "My Docker Runner" \
  --docker-image "docker:19.03.12" \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock


Gitlab runner, docker registry and self-signed certs I've set up a Gitlab registry with self-signed certificate. After that I had to convince docker app to trust my CA, I did this by putting CA.crt into

/etc/docker/certs.d/registry.gitlab.yourdomain:5000

But you have to convince docker executor and dind service in Gitlab Runner too, I've done this by passing cert through volumes in config.toml:

volumes = ["/cache", "/certs/client", "path/to/cert/ca.crt:/etc/docker/registry.gitlab.yourdomain:5000/ca.crt"]

See the problem? Gitlab Runner uses old volume specification, so :5000 part will break it. I know about existence of pre_clone_script, so that you can map your cert into some folder and then move it. But that will not solve the dind service problem - pre_clone_script will only be executed in 'main' container.

Oh well, the answer was simple: why do you need to use volume only for file transfer? Just volume the directory, containing problematic folder with cert in it.


Jenkins pipelines vs matrix vs multiple jobs for multiple branches across multiple platforms I've never done any CI work before, so I'm struggling to decide how to set up Jenkins as my first project at my new job.

From what I (sort of) understand:

Jobs are the generic work unit
Pipelines are like a job but are defined from  a series of steps from  scripts stored in a VCS repo
Matrix is like a job, but you can define axes to execute commands across

The basic setup for my project is 2 git repositories serving 3 platforms (one Windows, one Mac/Unix), each of which has 3 branches for different phases (so 9 builds total). The team wants something that will automate builds from each branch as things are committed to them, and then run some tests (which I'm also going to write). There are 2 machines dedicated to each branch. This will not include any kind of general deployment. It'll only be to the test machines.

I've set up a Jenkins master server that each branch stack connects to. I have 3 jobs set up at the moment targets a label I've assigned to the test machines, since I'm just testing with one branch for each platform. Each job uses the "run script" option, and contains something like:

# sets up the required authentication
$SLAVE_HOME/pre-build.sh

# then run the build script
./build.sh

# then a post-build/install script that automates installation of the packages,
# runs the product, and verifies it's started properly.
$SLAVE_HOME/install.sh

The scripts are unique to platform, and may eventually be unique to branch depending on if build steps change

I'll also need to integrate testing at some point - I'm planning to try using SikuliX for mac/unix and either that or Selenium+Winium for Windows. This will include aggregating results and build status to post on a not-yet-written monitoring page. That will add another couple platform-specific "global" steps.

Setting up 9 jobs that are very similar feels like I'm doing something wrong. I've considered a matrix per platform with branch axis, but I'll also need to report on build status/outputs, and having 6 outputs tied into one job might be problematic. Tests may also differ between branches as features are changed.

You can use the Jenkins pipeline (Jenkinsfile) to run the multiple branches in multiple stages, with the conditional 'IF'.
Below is an example from Jonico. I always use this example to create continuous delivery of monolithic  applications.
https://gist.github.com/jonico/e205b16cf07451b2f475543cf1541e70


K8s cluster in peered VPC can't reach RDS cluster I have two K8s clusters in the same VPC that I want to connect to an MySQL
Aurora RDS cluster. One of the clusters can reach the RDS cluster just fine.
The other, however, cannot. I'll call these eks-cluster-working and
eks-cluster-broken.
I have a security group allowing traffic to the cluster:

Type
Protocol
Port range
Source
Description

MYSQL/Aurora
TCP
3306
sg-1 (eks-cluster-sg-working)
This rule works

MYSQL/Aurora
TCP
3306
sg-2 (eks-cluster-sg-broken)
This one does not

Both eks-cluster-working and eks-cluster-broken have the same default
security groups that AWS creates when making an EKS cluster and they are both
on the same K8s version (1.18). The one exception is that eks-cluster-working
has an extra security group for the load balancer service since it's hosting a
web service and the other one (eks-cluster-broken) is not. Both clusters
have outbound traffic rules that can reach 0.0.0.0/0.
I have zero firewalls, firewall policies and network firewall rule groups set
up in the region the clusters are hosted in.
This is the Terraform configuration I have set up for peering from my VPCs
to the default VPC.
resource "aws_vpc_peering_connection" "default_to_environment" {
  count       = local.num_environment_vpcs
  peer_vpc_id = data.aws_vpc.environment[count.index].id
  auto_accept = true
  vpc_id      = var.vpc_default_id

  tags = {
    Name      = "Peer from default to ${data.aws_vpc.environment[count.index].tags["Name"]}"
    tf_module = "vpc"
  }
}

resource "aws_route_table" "r" {
  vpc_id = var.vpc_default_id

  # route = local.route_table_routes

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "igw-<number>"
  }

  dynamic route {
    for_each = [for r in local.route_table_routes : {
      cidr_block                = r.cidr_block
      vpc_peering_connection_id = r.vpc_peering_connection_id
    }]
    content {
      cidr_block                = route.value.cidr_block
      vpc_peering_connection_id = route.value.vpc_peering_connection_id
    }
  }

  tags = {
    Name = "default to environment VPCs"
  }
}

Expected result
I expect eks-cluster-broken to be able to connect to the RDS cluster just
like eks-cluster-working can.
Question
What can cause this? Any suggestions as to how to go about debugging this?

For my Peering Connection, I went to:
"Actions" -> "Edit DNS Settings" -> "Allow accepter VPC (vpc-<id>) to resolve DNS of requester VPC (vpc-<id-default>) hosts to private IP"
and checked it. (It was unchecked before). I had to wait a little while before this took effect.


Ansible - Azure resource referencing My aim is to have my vnet and subnet in a seperate resource group from the other parts of my infrastructure i.e Virtual Machines.
I have a short playbook that creates the resource group for my network (vnet & subnet), which works fine. I can build a VM via the portal and all is well.
My other playbook creates a resource group, creates a vmnic and VM. This playbook references the vnet and subnet created in the other resource group.
I'm encountering an error (see below) when referencing the resources in the network resource group I created. I found a previous post that has helped by suggesting the use of the "set_fact" module.
I've tried different ways of entering set_fact key values (full resource path / part resource path), but I keep getting an error. I suspect it maybe due to interpolation? Note the double "//" in the error message.
Could anyone provide some guidance on where I'm going wrong, please?
Error message:

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error creating or updating network interface test-vm-vnic1 - Azure Error: InvalidRequestFormat\nMessage: Cannot parse the request.\nException Details:\n\tError Code: InvalidJsonReferenceFormat\n\tMessage: Reference Id /subscriptions/REMOVED-SUB-ID/resourceGroups/test-rg/providers/Microsoft.Network/virtualNetworks//providers/Microsoft.Network/virtualNetworks/vnet-ansible-test/subnets//subnets/snet-ansible-test is not formatted correctly. The Id is expected to reference resources of type virtualNetworks/subnets. Path properties.ipConfigurations[0].properties.subnet."}

Part of playbook:
  tasks:
   - set_fact:
       vnet_rg_id: "/subscriptions/REMOVED-SUB-ID/resourceGroups/rg-ansible-test"
       vnet_id: "/providers/Microsoft.Network/virtualNetworks/vnet-ansible-test"
       snet_id: "/subnets/snet-ansible-test"

   - name: Create resource group
     azure_rm_resourcegroup:
       name: "{{ rg }}" # this is var declared earlier in the playbook
       location: "{{ location }}" # this is var declared earlier in the playbook

   - name: Create virtual network inteface cards for VM
     azure_rm_networkinterface:
       resource_group: "{{ rg }}" # this is var declared earlier in the playbook
       name: "{{ nic_name }}" # this is var declared earlier in the playbook
       virtual_network: "{{ vnet_id }}"
       subnet: "{{ snet_id }}"
       location: "{{ location }}" # this is var declared earlier in the playbook

With a bit trial and error I was able to work out the values for my "vnet_id" and "snet_id" variables.
I found using the full ID for the virtual network and not having a "/" before the subnet variable (snet_id) value fixed my problem.
At the moment I don't fully understand how Ansible/Azure puts it all together, but this has given me enough information to start to understand.
If anyone could point me to some documentation that explains the construction/interpolation of the variables it would be helpful.
  tasks:
   - set_fact:
       vnet_rg_id: "/subscriptions/REVMOVED-SUB-ID/resourceGroups/rg-ansible-test"
       vnet_id: "/subscriptions/REVMOVED-SUB-ID/resourceGroups/rg-ansible-test/providers/Microsoft.Network/virtualNetworks/vnet-ansible-test"
       snet_id: "snet-ansible-test"

   - name: Create resource group
     azure_rm_resourcegroup:
       name: "{{ rg }}" # declared earlier in the playbook
       location: "{{ location }}" # declared earlier in the playbook

   - name: Create virtual network inteface cards for VM
     azure_rm_networkinterface:
       resource_group: "{{ rg }}" # declared earlier in the playbook
       name: "{{ nic_name }}" # declared earlier in the playbook
       virtual_network: "{{ vnet_id }}"
       subnet: "{{ snet_id }}"
       location: "{{ location }}" # declared earlier in the playbook


What are the best practices for sharing production Postgres access with developers? We're a small SaaS startup (~10 person team; mostly engineers).
We use Postgres on AWS and GCP in production.
We have signed contracts with some customers that prohibit us from giving full production access to developers (to avoid security concerns). I think this is a good thing.
However, every now and then our backend team needs access to the database to analyze data, table records, and see how things are working.
What are some recommended best practices for this?
At some larger companies, they have internal tooling that gives developers access to a middleman CLI through which they can run SQL queries. This middleman logs these for auditing purposes.

We keep a RO clone of the important DB's.  Any dev can access the basic RO clone if he's in the right dev group vpn,  because the cloudsql instances are deployed in a private service net that peers with our private net and get be gotten to from the outside.  If not they can get added it a group..  Then they just need the ip address and they need the user and password.  There's an ROnly user given access to everything the postgres user gets, just RO.
PII data goes in a second PII DB, which is set up the same, but requires our compliance department to vet the person for pii access
We also have a firm policy of no poking the DB.  Look at it, sure, clone if you have to to determine how to fix a data error, but no raw manhandling.  It's to be done with a (possibly one shot) module off the codes db admin function.


Terraform creating single SG but multiple CIDR blocks I am trying to use Terraform to create a single security group that will allow traffic for mongo on port 27017 from all my private subnets that are stored in a variable. My issue is I am unsure how this would work since I only want one SG but the foreach creates multiple - at least it did before with subnets

my variable

variable "subnet_numbers_private" {
  description = "Useless info"
  default     = {
    "us-east-1a" = 1
    "us-east-1b" = 2
    "us-east-1c" = 3
  }
}

my security group resource

resource "aws_security_group" "mongo" {
  for_each = var.subnet_numbers_private
  name        = "Mongo"
  description = "Allow mongo traffic"
  vpc_id      = aws_vpc.Main_VPC.id

  ingress {
    from_port   = 27017
    to_port     = 27017
    protocol    = "tcp"
    cidr_blocks = [cidrsubnet(aws_vpc.Main_VPC.cidr_block, 8, each.value)]
  }

  tags = {
    Name = "Mongo-${each.key}"
  }
}

Currently this just provides an error that doesn't mean much to me as I don't know TF 0.12 well enough

Because aws_security_group.mongo has "for_each" set, its attributes must be
accessed on specific instances.

For example, to correlate with indices of a referring resource, use:
    aws_security_group.mongo[each.key]

Figured out the answer, fourtinately for me I had missed the example in the blog post about terraform 0.12 linked below

Here is my final (working how I want) security group

resource "aws_security_group" "mongo" {
  name        = "Mongo"
  description = "Allow mongo traffic"
  vpc_id      = aws_vpc.Main_VPC.id

  ingress {
    from_port   = 27017
    to_port     = 27017
    protocol    = "tcp"
    cidr_blocks = [
      for num in var.private_subnet_numbers:
      cidrsubnet(aws_vpc.Main_VPC.cidr_block, 8, num)
    ]
  }

  tags = {
    Name = "Mongo"
  }
}

Link to blog post about foreach - https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each/


Jenkins "internal" pipeline script producing errors when running sh steps I've been experimenting with creating a new pipeline, using the built-in pipeline editor (i.e. not from SCM). The strange bit is that, when I run any sh step, I receive the following error:

  sh: 1: Syntax error: end of file unexpected (expecting "done")

This comes from the following MWE:

pipeline {
    agent any

    stages {
        stage('Test') {
            steps {
                sh 'echo "FOO!"'
            }
        }
    }
}

The odd bit is that my pipeline scripts that are coming from SCM are working just fine, despite the fact that they have loads of shell scripts.

Running Jenkins 2.73.2.

Apparently this was caused by the pipeline name having an apostrophe character in it. Renaming the build solved the issue.


How can I diagnose php cli command execution time inside a docker container I've recently migrated an event subscription service called by a bash script from Ubuntu, into a Dockerised container.

The Docker image I am using is: FROM php:7.1-apache

The script itself is pretty straightforward:

#!/bin/bash

DIR="$(dirname $0)"
OUTPUT=""

while true
do
    OUTPUT=""
    OUTPUT=$OUTPUT$(php $DIR/../bin/cli.php bash:v1:issueEvents)
    OUTPUT=$OUTPUT$(php $DIR/../bin/cli.php bash:v1:processCustomerAccountEventSubscriptions)
    OUTPUT=$OUTPUT$(php $DIR/../bin/cli.php bash:v1:processProductionOrderEventSubscriptions)
    OUTPUT=$OUTPUT$(php $DIR/../bin/cli.php bash:v1:processWebEnquiryEventSubscriptions)
    OUTPUT=$OUTPUT$(php $DIR/../bin/cli.php bash:v1:processWebOrderEventSubscriptions)
    OUTPUT=$OUTPUT$(php $DIR/../bin/cli.php bash:v1:processWebQuoteEventSubscriptions)

    if [ -z "$OUTPUT" ]; then
        sleep 10
    fi
done

A complete round trip from event subscription to processing would take about 30 seconds on the host OS. Inside a container it is taking around 10 minutes.

My initial suspicion was that DNS was slowing things down, but I've installed dnsutils and the lookup times are the same both inside and outside the container.

I'd like advice regarding how I'd go about investigating the cause of this slowdown.

The general process of debugging inside the container is the same as debugging outside of one.  The only thing that you generally need to know is that you can use docker exec to attach a bash shell inside a running container, or override the command with --entrypoint to launch bash instead, and then run the script manually.

At that point, your question becomes "How do I debug slow execution times in a PHP script?".  That is not really in scope for this site, but you can start with a few basic things: add print statements or attach a profiler to see where things are getting slow, and then narrow it down from there.


If a piece of work requires manual testing/QA, do we need to wait until opening PR? First question here (i'm usually on StackOverflow)...not sure if the right place (if not, please vote to move where it belongs).

If we have a dev process using traditional GitFlow approach, what criteria should be met before a Pull Request (PR) is opened? Specifically, if the work requires manual testing/QA signoff before being merged into master.

There are two sides of the fence here:

PR's should not be opened until the work is "ready". (ready = ready for production, pending any code reviews, which should just be based on code quality/standards). Pros: reviewer doesn't need to check if the work is 'ready to be merged'. It's defined by the process. Cons: longer time (e.g waiting for test signoff) before the code is reviewed. PR's are open longer.
Open the PR, even if it's not ready. Basically, allow the QA testing & code reviews to operate in parallel. Pros: code review and testing can operate in parallel. Cons: if issues are found in testing, the code needs to change, even though it has already been reviewed. So reviewing could happen multiple times, and communication needs to be fed correctly between parties.

Thoughts on either 2 approaches, or is there an option 3? What standards/advice is there in this process?

The guidance that we've always followed in my various scrum teams is developers should get what they've got into testing as soon as possible. It may mean a little rework later on down the line if there are issues found in code reviews or developers haven't completely finished coding, but the earlier developers get feedback on what's been done, the earlier they can fix any issues, and take account of this feedback in any further coding.

You should make testers aware when they aren't getting completely finished output, but if they've been encouraged to think about the benefits of preventing bugs as well as just finding them, they should hopefully be keen to adopt this process.

It isn't always practicable - for example, if you don't have deployment tools that are flexible in which branches they deploy from, but when it is practicable, it's what Scrum and agile are all about - get feedback and get it early.


Include curl commands in a PostgreSQL Docker environment Using COPY table FROM PROGRAM 'curl "www.example.com/data.csv"'; I am able to copy a CSV directly from the web into a psql table.
However, when I am running the same command from inside a docker container I get the following error:
COPY sites FROM PROGRAM 'curl "www.example.com/data.csv"';
ERROR:  program "curl "www.example.com/data.csv"" failed
DETAIL:  command not found

Obviously curl is not available in the containerized environment. Therefore I changed my Postgres Docker image from image: postgres:12-alpine to image: postgres-12. Unfortunately I still get the same error.
How can include the curl command in a containerized PostgreSQL Docker environment?
This is my docker-compose file:
version: '3.7'

services:
  web:
    build:
      context: ./services/web
      dockerfile: Dockerfile.prod
    command: gunicorn --bind 0.0.0.0:5000 manage:app
    volumes:
      - static_volume:/home/app/web/project/static
      - media_volume:/home/app/web/project/media
    expose:
      - 5000
    env_file:
      - ./.env.prod
    depends_on:
      - db
  db:
    image: postgres:12
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    env_file:
      - ./.env.prod.db
  nginx:
    build: ./services/nginx
    volumes:
      - static_volume:/home/app/web/project/static
      - media_volume:/home/app/web/project/media
    ports:
      - 1337:80
    depends_on:
      - web
Include curl commands in a PostgreSQL Docker environment
volumes:
  postgres_data:
  static_volume:
  media_volume:

One way to use curl inside the container will be to use a custom Dockerfile to build you postgrsql container.
Just use the image: postgres:12-alpine as the base image of your Dockerfile and install curl in alpine with the following command : RUN apk --no-cache add curl


After removing Docker container, why is it automatically re-created? I'm using ubuntu-16.04 desktop and installed Docker-1.5-1 and docker-engine-17.05.0~ce-0~ubuntu-xenial, in which I installed openmedicus/phplist. My aim is to remove the container, and the image as well.

The issue is that the container is removed successfully but after some time the container is automatically created with a different container ID.

I even tried to remove images also but faced the same issue.

Trying to remove images:

$ docker images | grep phplist
openmedicus/phplist             latest              24d3de5279cb        4 months ago        694MB

$ docker rmi openmedicus/phplist:latest --force
Untagged: openmedicus/phplist:latest
Untagged: penmedicus/phplist@sha256:6ac98b2f624eb8cd61b64c4b398650975a876224d5925175c58ff0c1b05a2593

$ docker images | grep phplist    ## Delete Sucessfully

Now trying to remove container :

$ docker ps
b02b641560e8        24d3de5279cb             "/bin/sh -c /usr/s..."   2 minutes ago       Up 2 minutes        0.0.0.0:8080->80/tcp
$ docker stop b02b641560e8
b02b641560e8

$ docker rm b02b641560e8 --force   # Remove Sucessfully
b02b641560e8

Again, the container is created automatically :

$ docker ps
2c78199dc40c        openmedicus/phplist      "/bin/sh -c /usr/s..."   29 seconds ago      Up 28 seconds       0.0.0.0:8080->80/tcp                                 phplist

How can I remove the Docker container and image permanently?

Check the following:

Something like watchtower maybe running and monitoring your host machine on which the containers are running. Watchtower can re-pull your images from a registry and re-launch a container for you. There are some tools similar to watchtower which do this.


What are good options to implement a devops chatbot for Slack? I'm considering implementing a chatops platform for my company to help automate code deploys. I've seen some alternatives, which I narrowed down to cog and Errbot. Errbot seems to be actively maintained, while cog hasn't seen a commit to master since July 2018. I liked cog's architecture, which gives me a sense of performance and resiliency; it can easily be deployed in Kubernetes and it is extensible in any language. Also its ACL feature is a big plus.

My previous company's devops team used cog, and it seemed pretty solid to me. However, I'm concerned about deploying it as my current devops team is small and if cog is no longer maintained we cannot afford to maintain it.

Can you share your experiences with any chatops platform you're using, and what are their pros and cons?
I'm interested in chatbots that have at least these features:

Has an ACL built in, similar to cog
Able to be extended in any language (or at least its extensions can be implemented in Python/JavaScript).

I have found StackStorm to be the most fully-featured ChatOps product.
It satisfies your requirements:

Authz, RBAC, ACL ✔
Extensible in Python ✔

See below.

StackStorm is marketed as a

Robust If This, Then That Platform

(but for DevOps)
StackStorm consists of a few components, beyond it's chatops capability:

Sensors:
Sensors are Python plugins for either inbound or outbound integration that receives or watches for events respectively. When an event from external systems occurs and is processed by a sensor, a StackStorm trigger will be emitted into the system.

Triggers
Triggers are StackStorm representations of external events. There are generic triggers (e.g. timers, webhooks) and integration triggers (e.g. Sensu alert, JIRA issue updated). A new trigger type can be defined by writing a sensor plugin.

Actions
Actions are StackStorm outbound integrations. There are generic actions (ssh, REST call), integrations (OpenStack, Docker, Puppet), or custom actions. Actions are either Python plugins, or any scripts, consumed into StackStorm by adding a few lines of metadata. Actions can be invoked directly by user via CLI or API, or used and called as part of rules and workflows.

Rules:
Rules map triggers to actions (or to workflows), applying matching criteria and mapping trigger payload to action inputs.

Workflows
Workflows stitch actions together into “uber-actions”, defining the order, transition conditions, and passing the data. Most automations are more than one-step and thus need more than one action. Workflows, just like “atomic” actions, are available in the Action library, and can be invoked manually or triggered by rules.
Packs
Packs are the units of content deployment. They simplify the management and sharing of StackStorm pluggable content by grouping integrations (triggers and actions) and automations (rules and workflows). A growing number of packs are available on StackStorm Exchange. Users can create their own packs, share them on Github, or submit to the StackStorm Exchange.

Audits
Audit trail of action executions, manual or automated, is recorded and stored with full details of triggering context and execution results. It is also captured in audit logs for integrating with external logging and analytical tools: LogStash, Splunk, statsd, syslog.

StackStorm's ChatOps is built into it's other components:

ChatOps leverages two components within StackStorm in order to provide a fluid user experience. These subsystems are the Action Aliases and Notifications subsystems. You can learn more about each of these individual components in their corresponding sub-sections.

but you can also use your own:

If you already have a Hubot instance, you’ll need the hubot-stackstorm module installed and the following environment variables set up:

Authentication can be done by StackStorm itself, or delegated to a proxy authentication system:

The service can be configured with different backends (i.e. PAM, LDAP, etc.) to handle the authentication. If a backend is not specified, an htpasswd-compatible flat file authentication backend is used. To use a different backend, select and install the appropriate python package from the StackStorm community repos and configure st2auth accordingly.

It also supports API keys.
As for RBAC, they claim that it is supported in their "Extreme Workflow Composer"

Role Based Access Control (RBAC) is only available in Extreme Workflow Composer. For information about Extreme Workflow Composer and the differences between StackStorm and Extreme Workflow Composer, please see stackstorm.com/product.

More at https://docs.stackstorm.com/install/ewc.html
StackStorm has "packs" which are predefined workflows, similar to IFTT or Zapier, which are sets of triggers, rules and actions. You can extend it with your own actions:

An action is composed of two parts:

A YAML metadata file which describes the action, and its inputs.
A script file which implements the action logic

As noted above, an action script can be written in an arbitrary programming language, as long as it follows these conventions:

Script should exit with 0 status code on success and non-zero on error (e.g. 1)
All log messages should be printed to standard error


Is there a puppet module to manage policykit rule on Ubuntu 16.04? Is there any module which could help to manage policykit rules on puppet for Ubuntu 16.04 instead of putting file on /etc/polkit-1/localauthority/50-local.d ?

It appears that there is. For future reference, the best place to look for puppet modules generally is to search Puppet Forge. That is the central repository for modules and any module uploaded there can be installed using puppet by typing:

puppet module install jhoblitt-policykit

But obviously, you will want to update the name of the module to be whatever you want to install.


Why is ansible not sending the correct sudo password? What does it mean when this:

EXEC sshpass -d29 ssh -C -vvv
  -o ControlMaster=auto
  -o ControlPersist=60s
  -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no\
  -o Port=58030
  -o User=em7admin
  -o ConnectTimeout=30 127.0.0.1 /bin/sh
  -c 'sudo -H -p "[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: "
  -S -u root /bin/sh
  -c '"'"'echo BECOME-SUCCESS-mxraphooaxhpruunceorxkcelobceggf; LANG=C LC_MESSAGES=C LC_CTYPE=C /usr/bin/python'"'"''

returns this:

debug1: Sending command: /bin/sh -c 'sudo -H -p \"[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: \" -S -u root /bin/sh -c '\"'\"'echo BECOME-SUCCESS-mxraphooaxhpruunceorxkcelobceggf; LANG=C LC_MESSAGES=C LC_CTYPE=C /usr/bin/python'\"'\"''
debug2: channel 2: request exec confirm 1
debug3: mux_session_confirm: sending success reply
debug2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug1: mux_client_request_session: master session id: 2
debug2: channel 2: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
debug2: channel 2: read<=0 rfd 6 len 0
debug2: channel 2: read failed
debug2: channel 2: close_read
debug2: channel 2: input open -> drain
debug2: channel 2: ibuf empty
debug2: channel 2: send eof
debug2: channel 2: input drain -> closed
debug2: channel 2: rcvd ext data 67
[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: debug2: channel 2: written 67 to efd 8
debug2: channel 2: rcvd ext data 18
debug2: channel 2: rcvd ext data 67
Sorry, try again.\n[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: debug2: channel 2: written 85 to efd 8
debug2: channel 2: rcvd ext data 18
debug2: channel 2: rcvd ext data 67
Sorry, try again.\n[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: debug2: channel 2: written 85 to efd 8
debug2: channel 2: rcvd ext data 18
debug2: channel 2: rcvd ext data 36
Sorry, try again.\nsudo: 3 incorrect password attempts\ndebug2: channel 2: written 54 to efd 8
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug3: mux_exit_message: channel 2: exit message, exitval 1
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0

I've tried many combinations of ansible_ssh_pass and using the password from the commandline (I need to get ansible running so I can get the public keys on these devices),  but still it's sudo that isn't working.  Sudo does work for the username and password on the device.

The only wacky thing I'm doing here is setting the ansible_port after the play starts.

- hosts: devices
  gather_facts: False
  pre_tasks:
  - name: Get ephemeral port
    local_action: ephemeral_port
    register: ephemeral_port
  - name: Run local ssh command to enable proxy
    local_action: shell /usr/bin/ssh -p {{proxy_port}} -o 'HostKeyAlias=[{{proxy_ip}}]:{{proxy_port}}' -o 'UserKnownHostsFile=/home/user/.ssh/known_hosts' -i /home/user/.ssh/id_dsa -o 'ConnectTimeout=5' -o 'ControlMaster=auto' -o 'ControlPath=%r@%h:%p' -o 'ControlPersist=10s' -f -N -l user -L {{ephemeral_port.port}}:{{silo_private_ip}}:22 {{proxy_ip}}
  - name: Change ansible port to ephemeral port
    set_fact:
      ansible_port: "{{ephemeral_port.port}}"
  tasks:
    - name: test
      sudo: yes
      shell: hostname

I even had it working once, but I deleted the old playbook and I can't remember what was in it - this playbook is a lot simpler than the one I started with and I don't really want to start.

If I take out the pre_task part it works fine too (I just ran it once, copied the port number and hard coded it in as the ansible_port in the host_vars file)

The cure (or bandaid) was to set Pipelining=False in the ansible.cfg file.

This kind of stinks because I like Pipelining and it works, but I guess ansible is gonna treat that first ssh statement as something 'it owns' even though it's a local action.


Can a server used for GitLab runners be used as a deployment server? I have a relatively simple understanding of GitLab CI:

There is a GitLab server (server A)
There is a separate server with GitLab runners (server B)
And there is some other server - the 'prod' server (server C)

From prior experience, I understand that the GitLab runners would do the following:

On a push to source control, GitLab (server A) would contact the GitLab runner (server B), and the runner would pull the source control and execute jobs as defined in .gitlab.ci-yml, which is included in the source control
As defined by the .yml file, a Docker image would be added back to server A (to the GitLab Docker registry). Then the runner would login to server B, pull the docker image, and start it.

I am opting for a simpler automated deployment setup.. I want 2 servers:

Server A (GitLab CE)
Server B: Ubuntu 18 server that has a GitLab runner, AND that serves as a production server

Is this a good idea?

It seems like I can configure the runner to just run a docker-compose up --rebuild command, and that this would essentially work.

Is there any reason that I wouldn't want to do this?

GitLab and GitLab CI are not two servers: they are softwares installed on a server. Now, while both can be installed on the same server, they can be split for scaling reason.

Now your question: it all depends on what type of services your are running. Bear in mind that CI can take CPU/memory and disk bandwidth while running your CI (docker image build, composer, npm, Java, etc.). Doing so can slow down your production server.
Another reason not to run docker-compose directly is to avoid downtime: what if there is a mistake during the build or the deployment? Your production will be down. You may want to look at blue/green deployment.


Docker build not picking up built jar? I am new to the DevOps world but I am trying to build a pipeline using Jenkins and Docker.

I have Jenkins building the docker image by:

docker build -f Dockerfile --build-arg jenkins_jar_file=${JENKINS_JAR_FILE} -t ${DOCKER_IMAGE} .

Where JENKINS_JAR_FILE is a variable that points to the jar file produced by Jenkins

This is my Dockerfile:

FROM ~~container_on_private_registry~~
USER root
WORKDIR /root
ARG jenkins_jar_file
RUN mkdir -p /root/app/app.jar
COPY ${jenkins_jar_file} /root/app/app.jar
ENTRYPOINT ["java", "-jar", "/root/app/app.jar"]

However, I feel like the image is not being built with the right JAR?
I run docker images and see my image - but it says "Created 4 Days ago"
I see the same thing when I run docker images histroy for the entrypoint step.

When I actually try to run the image against a container, it for some odd reason tries to run against a jar which doesn't exist in the image. Instead it returns the following error:
/root/app-1.0-20180729.204127-14

Any help appreaciated :)

It seems to me that your Dockerfile should be looking like this:

FROM ~~container_on_private_registry~~
USER root
WORKDIR /root
ARG jenkins_jar_file=jenkins.jar
ENV jenkins_jar_file ${jenkins_jar_file}
RUN mkdir -p /root/app
COPY ${jenkins_jar_file} /root/app/app.jar
ENTRYPOINT ["java", "-jar", "/root/app/app.jar"]

Explanation:

We tell the Dockerfile to let Docker know, that it expects a variable named jenkins_jar_file to be passed to it during the build. Subsequent lines can reference that variable with a dollar notation.

Also when you build the image, use --no-cache so it will re-build all the layers of the image.


AWS ECR start uploaded docker image? I used packer to upload a docker image I created to my AWS ECR repository.

How can I use the AWS cli to automatically locate that image, create a new cluster, set the port rules to allow all inbound and outbound traffic and then spin up the container?
I want to automate this process so I can integrate it into packer later, as a post processor.

Can someone explain me how I can do this?

{
    "type": "docker-push",
    "ecr_login": true,
    "aws_access_key": "<snip>",
    "aws_secret_key": "<snipe>",
    "login_server": "https://<snip>.dkr.ecr.eu-west-1.amazonaws.com/"
}

Currently the last part of my packer script just uploads the image - now I need a way to automatically start this docker image and make sure the network traffic settings are configured correctly.

The answer depends on how you want to run the container.  There are 3 ways you could accomplish this:

Since you mentioned creating a cluster, you could create a new Kubernetes cluster and run your containers in this cluster.  AWS does support creating a cluster and managing of it via the CLI.
You can choose to run the containers as part of AWS Elastic Container Service (ECS).  This supports launching the containers via AWS Fargate (essentially a "serverless" type approach where you pay for CPU and Memory consumed) or on a standard EC2 instance.  ECS has its own CLI (separate install).  ECS will also allow you to set the ports and set inbound/outbound rules.
Finally, you can choose to use the CLI to spin up an EC2 instance and then manually or via a script start the docker containers.  However, you'd be much better off choosing one of the first two options.

Finally, AWS does have a tutorial that seems to go over exactly what you are trying to accomplish (configuring a container to run via the CLI).


1 second system engineering work cycles: educated workflow setup for Docker-driven development Maybe the whole setup is wrong but imagine your task is to model a system, or set of systems using Docker, where exact system configuraton can be tricky.

The workflow resulting from it seems like series of making small changes, waiting to analysing logs, changes, waiting... It seems like during an hour of work you could manage several iterations, but waiting is mind-boggling, though it already takes place on top of base images and inside-container work is done as well to try out things. Additionally, often this work can block many others.

Question: how to speed up this process? Do you guys use powerful appliances to reduce the cycle periods which is in term of lean manufacturing a clear waste given cost of systems compared to human time (imagine any change would be landing in the target environment always in just one second)? Or is the workflow wrong?

In some cases, depending on the actual configurations being targeted, it might be possible to try the effect of various changes live, using other automation tools, and only update the docker images (which acounts for a big chunk of that waiting time) after the bulk of the config changes are eliminated.

Stabilizing first the common/base docker image used across most/all of the candidate variants and sharing it would also help.


od output is confusing This isn't a deploy related question, but I believe it belongs here.
I've found out about the od tool yesterday, and began to fiddle with it.
The question is about utf-8 encoding. See the examples:
~ $ echo -n á | od -x
0000000 a1c3
0000002
~ $ echo -n áá | od -x
0000000 a1c3 a1c3
0000004
~ $ echo -n ác | od -x
0000000 a1c3 0063
0000003

# This is the issue
~ $ echo -n cá | od -x
0000000 c363 00a1
0000003

So á translates to c3 a1 correctly in each test besides the last, which begins with an ascii char. Why isn't the output of the last test 0063 a1c3 (i.e. invert the one before the last) ?
I've tried to read from a utf-8 file and the result is the same.
Thanks in advance.

Problem solved:
It's because of endianess of output.
There is an option --endian which by default is little.
The example string was cá, which translates to 63 C3 A1 utf-8 encoded:
~ $ echo -n cá | od --endian=big -x
0000000 63c3 a100
0000003

(The -n prevents \n in the end of string, making last byte 00)
So it's exactly the same output, but with bytes inverted:
~ $ echo -n cá | od --endian=little -x
0000000 c363 00a1
0000003

(c363 instead of 63c3 and so forth).