Passing replicas number into container I have a statefulset with an init container which creates openshift objects upon scaling. I need to pass it the replicas number from the same yaml definition.
However the replicas filed doesn't seem to be working with fieldRef. Is there another way to do it?

If you launch the statefulset with the following key set
{"spec":{"template":{"spec":{"serviceAccountName":"myserviceaccount"}}}}

you can run
curl \
https://kubernetes/apis/apps/v1/namespaces/<NAMESPACE>/statefulsets/<STATEFULSET> \
-k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
| jq '.spec.replicas'

to extract the replica count. This, of course, relies on the service account (myserviceaccount in this example) to have appropriate permissions to actually query the k8s API for this information, as well as having curl and jq available in the container.
You could then setup some kind of background job in the container to regularly make this request and update an env var or file or whatever.


How to create unique GPG Keys for different environments / projects After searching and reading various documents it remains unclear to me as to an effective/straightforward manner to create/maintain multiple sets of GPG Keys.

I have a project that needs unique GPG Keys per environment and each user has their own GPG Key. Thus each server (environment) needs unique GPG keys.

I would envision the most secure and best practice is for each environment would have its own gpg --gen-key results for which to base its own private and public keys. From there an admin could add a user to each environment as needed.

Or am I just missing the point all together?

If you want to create separate key pairs that are not dependent on your personal keys, the easiest way is to use a separate GPG home directory for each. For example, to create keys for "projectB", first create an empty directory, then tell gpg to use that dir as its home:

mkdir projectB-gpg
gpg --homedir ./projectB-gpg --gen-key

That will do what you want, although you will now have the problem of distributing the private keys and passwords securely to each environment.


Communication between VPCs and a transit VPC hosting VPN Server without Masking source IP I am setting up a VPN Server to move all resources behind VPN on AWS, which includes moving multiple VPCs.

The simple setup is:

I have a separate VPC (VPC1) running VPN Server and another VPC (VPC2) with an EC2 machine.

Let's assume the following CIDRs for VPCs & VPN.

VPC1: 10.1.0.0/16
VPC2: 10.2.0.0/16
VPN: 10.8.0.0/24

Everything is working fine within VPC1.

Now I have enabled VPC Peering between  VPC1 & VPC2, but I am not able to access EC2 machine running in VPC2 unless I mask the VPN CIDR as following.

iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

But doing so, I lose the source IP of the request.  All requests are forwarded as requested by VPN machine.

Let me know if any other information/clarification is required.

VPC peering does not support transit traffic.

Unsupported VPC Peering Configurations described several scenarious that are not supported.

This case is not specifically mentioned, but is implicit from the other limitations.

Note also that it is impossible to fully configure the setup you are attempting, because when you route a given destination towards a remote VPC via a peering connection, there is no route table in the target VPC that is applicable to that traffic. Only the instance subnets are available.  Route tables apply only to subnets, and the default route table only applies to subnets without an explicitly assigned route table -- not to incoming traffic from a peering connection.  There is no way to route this traffic back to your instance.

This is why Hardware VPN, Direct Connect, and NAT devices are not available across peering,

A Transit VPC is not what you have described.  These use AWS VPC Hardware VPN connections on the spokes and an EC2-based router at the hub to terminate the tunnels... and the transit routing between the tunnels is done within the instance -- not by the VPC network infrastructure.


CI platforms versus simple git hooks Newbie to devops here and apologies this is probably a stupid question, but, what is the point of CI platforms? It seems to me that just by using git hooks I can automatically run all my unit tests before a commit and block the commit if needed. If I need to run tests in a production clone environment (I don't, but for argument sake), then I should be able to do this by spinning up a clone server and using git server hooks no? It's just that every article I'm reading on CI seems to imply that a CI platform is necessary. But so far nothing has said why. What essential component of CI can I not get unless I use Travis/Circle/Jenkins or whatever?

The git hooks approach could probably work fine for lone developers. But when multiple developers work on the same project it doesn't work so well, especially in large scale projects.

Several reasons:

the git hooks are local to your copy of the repository, there is no check for consistency across all developers. Anyone can "adjust" their hook script as they see fit, or even forget to get them set up (they're not automatically installed when cloning git repositories for example). A CI platform will consistently run the same scripts for all changes.
the verifications performed are typically done on the developer's computer, so you're right into the typical "it worked for me" case. Again - consistency: a CI platform will perform the same verifications, in the same context, on a pool of resources much easier to keep consistent than developers' environments
the verifications performed by a developer are fundamentally silo'd, performed in isolation, on the local snapshot of the repository branch, before the changeset is actually integrated into the branch - they do not take into account the changes from the other developers which can happen at any time. Results obtained that way could be significantly misleading, see How can successfully pre-verified changes cause regressions which should have been caught?. CI platforms focus on the repository branch itself, checking how a particular changeset performs on that branch at that particular moment, with all other changesets already taken into account, reflecting the actual branch integration status.
(in larger projects) a CI tool can perform verifications which are impractical for the individual developers. For example multi-platform build lasting hours and requiring high performance build machines well above the typical developer laptop. The centralized verifications performed by a CI platform can be much more resource frugal than when performed by each individual developer.

Side note: not all projects benefit from the developer ability to spin up on demand a production-like environment/clone server ;) But even if possible, most of the above reasons still stand


What is the boundary of DevOps engineer job role? I will not get into what is DevOps since everyone have their own meaning.

But I would like to understand what is the boundary of DevOps engineer. Sometimes people ping me and ask to me that,

  can you please help me setting up VPN in my laptop?

sometimes,

  From the RDP machine, I am not able to open MySQL workbench, it hangs,
  can you fix it please?

sometimes,

  Production application is having performance issues, why is it
  happening?

sometimes,

  Can you please schedule the performance testing script?

sometimes,

  can you setup the spark and hadoop cluster?

sometimes,

  can you create users in MySQL?

I think we need an industry wide boundary, otherwise, the role of DevOps engineer is going in every direction.

From the "how it should properly work"/"how it was meant" perspective there is no real "DevOps engineer" because DevOps is a methodology/culture which emphasizes also on technical practices which should stretch across as many roles as possible and feasible.

How can we apply the CALMS (Culture, Automation, Lean, Measure, Share) mantra here? (Taking into approach that depending on factors like maturity, complexity and resources in the context, approaches need to be adapted to what helps to solve problems in an efficient and sustainable way)

Requirements engineer: can I automatically qualify maturity of requirements and track their delivery?
Software engineer: what is required to run my software in test either productive environment?
Architect: can I have a big picture based on live metrics?
Tester: can I create virtual temporary infrastructure to support execution of my code?
System administrator: how can I better organize my help desk / 1st user support and leverage self-help / automation solutions?

Somebody with a wish to focus on the methodology itself will be somewhere in-between all these guys and obviously become "jack of all trades, master of none".

Given this, my consideration how I would approach your customers' inquiries could be:

Please follow the instructions to setup VPN in this Wiki/code from this Git repo and if it's not working, document/show steps you have taken.
Please share more details on the context (here is not clear who owns which parts).
Let's allocate time to develop emergency plan for immediate action if required and also run scaling tests on a test system to model experiments, collect data and finally learn under which circumstances the performance problems appear. Then, we will know why it is happening. :-)
Follow from 3.? What does mean schedule from your perspective and if you mean regular execution, how often? So please let's go back to previous item :-)
Possibly, have you got more specs which infrastructure option (Cloud, on premises) should we use for that and what and what are you going to do? Here is a link to generic infrastructure planner, let's allocate time to discuss what is specific for Spark/Hadoop here and how will delivery to it look like.
If identity management is not part of our architecture, let's assess what are you going to do. An outcome could be that you include creation of users to your database scripts.

So the bottom line in DevOps culture regarding the role boundary is:

professional communication first
the boundary is to set boundaries inviting people to a professional conversation between adults, to get more conscious on both sides (!) about requirements, responsibilities and goals. Not just thinking in solutions and delegation but also make a plan and assess risks.

Therefore seasoned and successful professionals who are today maybe 50+ yrs old say sometimes jokingly "we have been doing DevOps from the very beginning" and partly, they are right.

To conclude and highlight and reflect on the real-world existence of such job-role, organizations who do not have a DevOps culture proceed to define DevOps roles as the following, for example:

DevOps engineer is an advanced systems administrator who can code up the infrastructure and automate it completely.
DevOps engineer is a tools administrator with coding skills to support the Dev team and ensure delivery towards production (CI/CD, deployment scripting but also Jira administration)
DevOps engineer is a developer focusing on including infrastructure elements into development, like containers and cloud APIs.

After all, many companies still look for the "full stack engineer" who should obviously become additionally the DevOps engineer (along with "AI/ML,  blockchain and foobar").

Finally, the role boundary, possibly not only in the DevOps context, but in this special context indeed very likely:

is also in your responsibility
may change between projects and employers or just from time to time.


Azure Devops Pipeline variables in Powershell I've tried to figure this out and I can't see why my ipaddress is not getting set.
I've checked code samples and they all suggest I am doing everything correct.
ip variable defined here:

This is my powershell script:
# Write your PowerShell commands here.

Write-Host "AGENTIPADDRESS variable is set to ($env:AGENTIPADDRESS)"

$ipaddress = Invoke-RestMethod http://ipinfo.io/json | Select -exp ip
Write-Host "ipaddress variable is set to ($ipaddress)"
Write-Host "ip pipeline variable to ($ip)"

Write-Host "##vso[task.setvariable variable=IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=env:IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=$env:IP;]$ipaddress"
echo "##vso[task.setvariable variable=IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=IP;]333"
Write-Host "##vso[task.setvariable variable=IP;]$ipaddress"
echo "##vso[task.setvariable variable=IP]999"
echo "##vso[task.setvariable variable=IP;]888"

Write-Host "##vso[task.setvariable variable=IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=IP]114"
Write-Host "##vso[task.setvariable variable=IP;]123"

Write-Host "ip environment variable to ($ip)"
Write-Host "env:IP environment variable to ($env:IP)"

Write-Host "Set environment variable to ($env:AGENTIPADDRESS)"

And here is my output:

Why can't I set & read the ip variable?

The issue is that you can't reference the changed value in the same task.
I created a new Powershell task and just called Write-Host "env:IP environment variable set to ($env:IP)" which output the IP address correctly.


What are the differences between application virtualization, emulation layer, and container virtualization? What are the differences and relations between application virtualization (such as Flatpak), emulation layer (such as wine, cygwin), and container virtualization (such as Docker)?

I have been trying to figure them out, but always end up not catching the main points. Especially the fundamental differences between their virutalization levels.

Container virtualization with Linux is a combination of kernel features, namespaces and cgroups, used to run an application in a sandbox. Each application has it's own filesystem, network stack, pid namespace, etc, but runs on the same OS kernel as the host. From the host you can see the processes running, but from inside the sandbox, all you can see is your application. It is designed for managing server side applications.

Application virtualization is very similar to container virtualization, and the lines are a bit fuzzy to me. In the context of something like Flatpak, it is using the same namespaces and cgroups you see with container tools like docker. The key difference is they have designed the tool for desktop use cases, meaning tools that are designed to be accessed with a local GUI, rather than running remotely as something like a web server.

Emulation is a way to run programs designed for another OS. You typically cannot run a Windows binary on a Linux system and vice versa, the OS kernels are very different, the binary formats are different, they're effectively speaking different languages. The emulation layer provides a translation to run non-native programs on the local machine without creating an entire virtual machine and running a separate OS. This is very different from the container and application virtualization where you run native OS applications on a shared OS kernel.


Flask, Gunicorn and Docker Swarm so I have this nice rest API implemented in Flask running in docker and I am thinking about scaling. This is how I progressed:

CMD ["flask", "run", "--host=0.0.0.0"]

and

docker run -d -p 5000:5000 pyrest-alpine

works fine

CMD ["gunicorn", "-w", "4", "-b", "0.0.0.0:5000", "pyrest:app"]

and

docker run -d -p 5000:5000 pyrest-alpine

This also works fine, Gunicorn provided some scaling via prefork worker model in a single container now I want to scale via replication of Docker swarm with healthchecks where both

CMD ["gunicorn", "-w", "4", "-b", "0.0.0.0:5000", "pyrest:app"]

and

CMD ["flask", "run", "--host=0.0.0.0"]

work just fine with

docker service create --name pyrest-swarm --replicas 2 -p 5000:5000 --health-interval=2s --health-timeout=10s --health-retries=3--health-cmd "curl 0.0.0.0:5000/status || exit 1" pyrest-alpine

My question is Do I still use gunicorn when using docker swarm replication?

Yes, when using Docker Swarm you'll still want to use Gunicorn in your Docker containers.

Gunicorn is necessary to facilitate the communication between the server and your web application. In the case of a Dockerized Flask application this is still the case as Gunicorn handles the communication between the Flask application and the Docker container. Adding Docker Swarm to this stack basically just means that you have more instances of your container, and that Swarm will handle balancing the incoming requests to each instance.


Terraform | After upgrading to 0.12 receive a lots of error "unsupported value for "%s" at 0: string required." After upgrading to 0.12 I receive a lots of error like:

Error: Error in function call

  on main.tf line 136, in data "template_file" "inventory":
 136:       formatlist(
 137:
 138:
 139:
 140:
    |----------------
    | module.mongodb.instance_names is tuple with 1 element
    | module.mongodb.private_ip is list of string with 1 element

Call to function "formatlist" failed: error on format iteration 0: unsupported
value for "%s" at 0: string required.

Here is from my main.ft:

connection_strings_mongodb_standalone = join(
      "\n",
      formatlist(
        "%s ansible_host=%s ansible_user=ubuntu",
        module.mongodb.instance_names,
        module.mongodb.private_ip,
      ),

And module:

module "mongodb" {
  source                      = "../modules/mongodb"
  instance_count              = "1"
  environment                 = var.env
  instance_type               = "i3.large"
  subnet_id                   = element(tolist(data.aws_subnet_ids.private.ids), 0)
  associate_public_ip_address = false
  vpc_id                      = data.aws_vpc.vpc.id
  tags = merge(
    {
      "Environment" = var.env
    },
    var.default_tags,
  )
  iam_instance_profile = var.iam_instance_profile
  route53_zone_id      = data.aws_route53_zone.zone.zone_id
  ami                  = var.ami
}

output "mongodb_private_ip" {
  value = module.mongodb.private_ip
}

output "tags" {
  value = module.mongodb.instance_names
}

from ../modules/mongodb output.tf:

output "instance_names" {
  value = [
    "${data.template_file.instance_tags_name.*.rendered}",
  ]
}

As I correct understend need to set type of output value, but can't find how, hope someone can help me.
Thanks in advance.

There is a clue in your error message, but it's only helpful if you know what you're looking for:

    | module.mongodb.instance_names is tuple with 1 element
    | module.mongodb.private_ip is list of string with 1 element

Your declaration of the instance_names output includes this expression:

["${data.template_file.instance_tags_name.*.rendered}"]

The data.template_file.instance_tags_name.*.rendered part of this returns a list of strings, and then the [ ... ] around it then wraps an extra list (or rather, tuple) around it, creating a single-element tuple containing a list of strings. That's why the tuple contains only one element, rather than having an element for each of your count instances.

This situation is what the upgrade guide discusses under Referring to List Variables, and that documentation includes some more background information on why this previously worked in 0.11 and why the change is required now.

To fix this, you should refer to just the splat expression alone, producing the flat list of strings that your formatlist call is expecting:

output "instance_names" {
  value = data.template_file.instance_tags_name.*.rendered
}

The automatic configuration upgrade tool described in the upgrade guide can make this change automatically in some cases, so I'd recommend using that as a starting point for your 0.12 upgrade process. It isn't able to fix everything, but it will do a lot of the straightforward rewriting work for you and will emit warnings about some more complex situations it's unable to handle.


What is "Push on Green"? Within Google and other DevOps organisations, they talk about "Push on Green" which I am lead to believe is the practice of continuous delivery based upon a successful run of automated tests, and presumably deployments to pre-live environments.

What exactly does "Push on Green" and what are the implications of applying it?

I Googled "Push on Green" and the first link was:
https://www.usenix.org/conference/lisa14/conference-program/presentation/klein

This was representative of almost the entire first page.  It looks like this term originated in Google's SRE group and has been taken up by the industry at large.

You are correct- "push on green" means that deployments are automatically executed when all tests pass.  This could be considered synonymous with "Continuous Deployment."  The accepted method for automating deployments to environments is to automate testing.

If your automated testing is rigorous enough, you can deploy any code for which the tests pass.  In very mature enterprises (the DevOps unicorns of all shapes and sizes) this can lead to Continuous Deployment all the way to the production environment.

My personal experience is that reaching this level of automated testing is a challenge in any enterprise.  I am currently in the process of pursuing this level of maturity in my current position.  Our first major milestone will be "push on green" deployments to any environment beyond "Integrated Development" (our "lowest" environment.)

This challenge is both technical and cultural.  As an IT organization, we owe it to our business stakeholders to prove that our testing is in fact rigorous enough to serve as the sole gateway to a deployment.  Once we have done that to our own satisfaction, we have to convince those stakeholders to let us try Continuous Deployment to the next environment (for us, this is QA.)


Is it possible to keep all the DSC configurations in Version Control and make Azure Automation fetch it from there automatically? The Azure Automation DSC seems to know to integrate with GitHub, but it seems to expect only runbooks there. At least, this is what I understand from the following UI screen:

I have just started to learn DSC and have no runbooks. So far I only used DSC configurations, no runbooks.

So, is it only possible for runbooks? Does it mean that nobody works with just DSC configurations?

P.S. Knowing that DSC stands for Desired State Configuration I find the term DSC configurations a bit funny.

The article "Continuous Deployment to Azure Automation DSC", or series of articles, walks through setting up a CI/CD pipeline for DSC using VSTS (Visual Studio Team Services).  Building a VSTS account in Azure is free and it provides some number of build minutes for free as well.

Basically, rather than setting up a runbook, you can configure a CI/CD pipeline using build services in VSTS to perform whatever orchestration activities you require for DSC and VSTS enables you to save your code in their source control management system.


"Error: forwarding ports: Upgrade request required" Error in helm of a kubernetes cluster I have a kubernetes cluster built using kubespray and imported to Rancher.

The nodes are configured with

CentOS Linux 7 3.10.0-957.12.1.el7.x86_64
Docker version : 18.9.5
Kubelet version : v1.14.1
Tiller version : v2.14.1 ( got this version from the tiller pod's image gcr.io/kubernetes-helm/tiller:v2.14.1 )

All the tiller resources are working fine:

$ kubectl get all -n kube-system | findstr tiller

pod/tiller-deploy-57ff77d846-frtb7             1/1       Running   0          12d

service/tiller-deploy          ClusterIP   10.233.49.112   <none>        44134/TCP                16d

deployment.apps/tiller-deploy             1         1         1            1           16d

replicaset.apps/tiller-deploy-57ff77d846             1         1         1         12d
replicaset.apps/tiller-deploy-69d5cd79bb             0         0         0         16d

But when I run the helm commands, I am getting this error:

$ helm version

Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Error: forwarding ports: error upgrading connection: Upgrade request required

$ helm ls

Error: forwarding ports: error upgrading connection: Upgrade request required

I tried:

The tiller version is 2.14.1. So, upgraded the helm client to version 2.14.1 from 2.11.0. But that doesn't solves the issue.

Can someone help me to solve this error?

See my answer here:

Ran into this today when trying to use Garden.io for a cluster running in Jelastic.
Found the solution in this Github comment:
First acquire a local binary for Tiller (server-version of Helm), either by compiling or by downloading it from the release page.
Then run:
$ export HELM_HOST=":44134"
$ tiller -listen ${HELM_HOST} -alsologtostderr >/dev/null 2>&1 &

This will run a local version of the Kubernetes Helm Server. Now try your original command again, kubectl, that will delegate to this local Helm instead and manage to connect.


How to access local Docker storage on Windows 10? Where does Docker daemon store image and container data on the local disk ?

Most of the persisted data is written to:
C:\ProgramData\docker

you should have directories here such as:

.\windowsfilter
.\volumes
.\config
.\image

etc

The layer and image data is stored under these.


GCP compute quota hard caps? Are there published upper limits or hard caps for the GCE CPU or Managed Instance Group Quota per account? I know that you can request to increase your Quota, but I have not been able to find what the limits are for these requests.

There is a Quotas page in the GCP console.  This Google help doc will tell you more and how to go about identifying your limits and increasing your quotas: https://cloud.google.com/compute/quotas


how to push an image to docker hub? How do I push this image to docker hub?
root $
root $ docker images helloworld_web
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
helloworld_web      latest              3118807ab0e0        5 minutes ago       196MB
root $

It's just the sample from docker.

horribly insecure:
root $
root $ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: nicksaunders
Password:
WARNING! Your password will be stored unencrypted in /root/snap/docker/471/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
root $
root $ docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
helloworld_web               latest              3118807ab0e0        13 minutes ago      196MB
frappe/erpnext-nginx         latest              e417869e11ba        39 hours ago        414MB
frappe/erpnext-worker        latest              96559438dfec        40 hours ago        1.22GB
frappe/frappe-nginx          latest              3d006c375c6e        40 hours ago        410MB
frappe/frappe-worker         latest              961988528953        40 hours ago        1GB
frappe/frappe-socketio       latest              bca49f1b4a04        40 hours ago        180MB
frappe/bench                 latest              8c1bc9d3a572        40 hours ago        1.4GB
frappe/erpnext-docs-nginx    latest              7d117e2bd798        42 hours ago        428MB
frappe/erpnext-docs-worker   latest              a1c0b0511e92        42 hours ago        2.88GB
frappe/erpnext-worker        <none>              09be9dde8f8f        2 days ago          1.21GB
frappe/bench                 <none>              c775a49dee59        2 days ago          1.4GB
redis                        alpine              bd71e6db4a54        13 days ago         32.2MB
python                       3.7-alpine          295b051ee125        2 weeks ago         41.7MB
basex/basexhttp              latest              3339bf6bc898        3 weeks ago         226MB
twilio/twilio-java           latest              611ab7ece1cf        5 weeks ago         993MB
mysql                        latest              0d64f46acfd1        7 weeks ago         544MB
jetty                        latest              5f997007f18d        7 weeks ago         522MB
jitsi/web                    latest              1fad75476320        2 months ago        483MB
mysql/mysql-server           latest              8a3a24ad33be        2 months ago        366MB
hello-world                  latest              bf756fb1ae65        8 months ago        13.3kB
lukptr/erpnext7              latest              130f3d9f222f        14 months ago       3.59GB
root $
root $ docker tag helloworld_web nicksaunders/helloworld:firsttry
root $
root $ docker push nicksaunders/helloworld
The push refers to repository [docker.io/nicksaunders/helloworld]
fca41575e369: Pushed
40f0977ea685: Pushing [==================================================>]  12.08MB
f8b013a4d861: Pushed
1d6d0bace1b8: Pushing [====>                                              ]  13.03MB/143MB
aabee2d268fc: Pushed
22ee430ae506: Waiting
60faa6c61cf9: Waiting
cfd52085ef94: Waiting
408e53c5e3b2: Waiting
50644c29ef5a: Waiting

but seems to work.


Ansible - private git repositories - SSH agent forwarding vs copying private SSH key I recently started playing around with Ansible and it seems very nice. I don't have much experience in DevOps stuff and never really had to handle any complex scenarios. I started creating my Ansible playbook to replace my current deployment tool - Deployer PHP. I'm stuck at cloning git repository unfortunately. Now, I know I need a public key added to enable access to git repository and here comes my question.

Should I be using SSH agent forwarding (this way I can use my local SSH keys) or should I store private SSH key (encrypted, added to source control) within my ansible project and copy it using Ansible to my target node? I know the question may be very broad, so what interests me is security implications of both approaches.

You can also consider to encrypt the file containing your private SSH key using Ansible Vault.

Some links:

http://docs.ansible.com/ansible/latest/playbooks_vault.html

https://therealmarv.com/ansible-vault-file-handling/

https://www.calazan.com/how-to-deploy-encrypted-copies-of-your-ssl-keys-and-other-files-with-ansible-and-openssl/


Test coverage of changed code in pull requests in git-tfs We have a lot of legacy code that have poor test coverage. We wish that all new code that is checked in through pull requests has (close to) full test coverage. What is the best approach or tools to achieve this?

We're using git-tfs, .NET C# and TypeScript. And we'd like this to be part of the code review process and prevent developers (or at least give a warning) to check in with insufficient test coverage.

IMHO from the workflow prospective code coverage measurements are fundamentally similar to builds, smoketests or any other types of QA checks and can thus be performed in similar manner. I can see several options:

include them in the PR verification path, just like builds and/or smoketests, similarly positioned relative to the code review stage.
include them as a step in the pre-commit gating system, if you use one - most effective IMHO, especially for large projects where scalability comes into play (mentioned in my answer to How do you scale your integration testing?).
include them in the CI pipeline, maybe on a parallel execution path if they're lengthy, to not add too much to the total CI execution time. Very useful to use these runs as up2date baselines for results comparisons for the other 2 cases listed above.

I'm not familiar with MS tools, but if MSTest doesn't offer good integration with the other tools in your workflow you could build custom wrappers to implement a typical submit/execute API with a pass/fail result (or set of results) which most tools can integrate with.


Bamboo equivalent to Jenkins color I'm currently building out a radiator screen to bridge a few different areas of our CI\CD pipeline, pulling in metrics as well as build statuses. We're also migrating from Jenkins to Bamboo

In Jenkins, when you do a rest call to the job/api/json it, one of the response fields is a color, which acts as an initial health. Great for colour coding, especially when things go red!

However, as far as I can see, there isn't a bamboo equivalent. The two calls I'm using so far in bamboo are

/result/{projectKey}-{buildKey}/{buildNumber : ([0-9]+)|(latest)}

/result/status/{projectKey}-{buildKey}-{buildNumber : ([0-9]+)}

Any clues or pointers as to where I can get the info from?

p.s. Writing this app in React \ Node

Bamboo doesn't have a color status like Jenkins per say, but it does have a very similar build status indicated by a circle with a symbol within it. This status is available for both the entire build plan and for an individual build.

For your GET request, you will want to use the same api calls that you are using to get the buildstate value to get a SUCCESSFUL or FAILED state.


Forwarding email for an entire domain I recently acquired the .com of my company name; previously been using .net domain.

I don’t want to switch emails from  .net, is it possible to setup anything at my .com to forward to my .net transparently?

I.E. somebody emails sales@mycompany.com and it forwards the message to sales@mycompany.net keeping the original sender name and address.

depending of the registar and where your email server is you can. For example Ionos (ex 1and1) allow user to redirect any un-existing email to another email. This is done with the wildcard *@yourdomainname.ext


What are reasons for using HashiCorp's packer to build docker images instead of using docker build? I have read https://www.packer.io/docs/builders/docker.html, but I do not see the advantage of using Packer over docker build or docker-compose up --build for building docker images

Mainly the reason is to keep your image building steps intact if you get to move from docker to another image building system.

Packer does support a bunch of providers (builders in packer terminology), and changing the target "container" is just a matter of changing the builder (or using multiple builders in the same packer file), the build steps (provisionner step) are kept intact and will be the same if you build a docker image or an AWS ami for example (you can even build both at the same time).


Tomcat Restart from jenkins I want to stop tomcat using jenkins. Tomcat is running as a seperate application server, I want to restart it using Jenkins console.

It’s possible to use Publish Over SSH Plugin. Lets set up ssh connection details in the settings interface of Jenkins. We assign server a name, an address, user’s name and authorization way — by password, passphrase orindicatedcate key. These adjustments are simple and easy. You may found more details the on plugin’s page.

After Install Plugin Add remote server

After Add remote server you can find in Build section Send file or Execute command over SSH

Here you can write command for Tomcat start/stop/restart

for more details visit this URL: https://medium.com/@weblab_tech/how-to-publish-artifacts-in-jenkins-f021b17fde71


How to store/retrieve multiple ansible-vault passwords within/from a single file Update: As @yaobin pointed out. It appears they removed the part about the multiple passwords being in a single file. It was here as can be seen from web archive:
https://web.archive.org/web/20201202040251/https://docs.ansible.com/ansible/latest/user_guide/vault.html
But now it is gone:
https://web.archive.org/web/20210128225633/https://docs.ansible.com/ansible/latest/user_guide/vault.html
So I am not sure if this problem can be solved anymore the way that was described in the manual at the time. It was apparently a manual bug :(

The manual says at storing passwords in files section:

To store a vault password in a file, enter the password as a string on
a single line in the file. Make sure the permissions on the file are
appropriate. Do not add password files to source control. If you have
multiple passwords, you can store them all in a single file, as long
as they all have vault IDs. For each password, create a separate line
and enter the vault ID, a space, then the password as a string. For
example:

and there is an example:
dev my_dev_pass
test my_test_pass
prod my_prod_pass

However there is no usage exampes for this file. I tried many different ways. If I use --vault-password-file the ansible-vault sees only default entry independent of the number of entries in file. With --encrypt-vault-id=test I get:
ERROR! Did not find a match for --encrypt-vault-id=test in the known vault-ids ['default']

If I use --vault-id abc@file_path with --encrypt-vault-id=test then I get error
ERROR! Did not find a match for --encrypt-vault-id=test in the known vault-ids ['abc']

What is the purpose of the password file described in the Ansible manual and how does one use it?
Thanks!

I totally misunderstood you. Here is a new attempt to actually answer your question. :)

Create a passwords file:

root@2ca77340d571:/# cat .passvaults
dev my_dev_pass
test my_test_pass
prod my_prod_pas

Create a file to encrypt:

root@2ca77340d571:/# cat extravaulty.yml
chocolate:banana

Encrypt the file:

root@2ca77340d571:/# ansible-vault encrypt --vault-id=dev@.passvaults extravaulty.yml
Encryption successful

View the raw file:

root@2ca77340d571:/# cat extravaulty.yml
$ANSIBLE_VAULT;1.2;AES256;dev
33336336336238326137393764316333623231336238323931306166626434653164326330656566
3561356333343435663538623661363661373461356461300a663361666331323031623530343930
62633230636566643339663637386438336539383162346634393031633165333033396238363163
6134383233346538330a326266626261346638303738313862656337396237343231326231656139
34376334663664343164303462356431306539316232323761386164643735376330

and the decrypted file:
root@2ca77340d571:/# ansible-vault view --vault-id=dev@.passvaults extravaulty.yml
chocolate:banana

As I understand it, when you use --vault-id=dev@.passvaults you are telling it to use dev as a hint (not the vault password) and .passvaults as the file where the encrypt/decrypt password is.
This means that you can also do this:
root@2ca77340d571:/# ansible-vault view --vault-id=prod@.passvaults extravaulty.yml
chocolate:banana

Or this
root@2ca77340d571:/# ansible-vault view --vault-id=.passvaults extravaulty.yml
chocolate:banana

Which is not intuitive to me.
Does this help?


How do I include my internal CA certificate to validate SSL exchanges in Chef? We do use an internal Certificate Authority to create server certificates in my company.

We also have to deal with a transparent proxy doing SSL interception (MITM).

I regularly encounter SSL validation errors due to Chef not knowing the CA  certificate, and sometimes it's the tooling around chef itself (berkshelf, knife; even chef client itself when talking to the server for the first time since chef 12 enables SSL by default).

Question is: How do I make Chef aware of my CA certificate to get valid SSL exchanges ?

There's a couple of way to achieve the result:

Chef has a trusted_dir to allow adding certificate to the trusted list. the documentation has a lot of details about it. Adding your CA certificate to this directory would solve the problem. knife has it also in a slightly different path as per it's own documentation
Chef use its own CA certiticate list in /opt/chef/embedded/ssl/certs/cacert.pem.
You can add your CA certificate at end of this list to trust it.

The second option has an advantage of allowing you to export the environment variable SSL_CERT_FILE pointing to chef cacert.pem to allow most of the tools using openssl library to know your CA certificate.

For the case of a self signed certificate on the chef server (or another server used as target in a recipe), knife ssl_fetch would allow all knife commands to work.

To add the server certificate to the cacert.pem for the case 2. above, you can use the following command:

# For a self signed CA certiticate
openssl s_client -showcerts -connect <YOUR_CHEF_SERVER>:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >> /opt/chefdk/embedded/ssl/certs/cacert.pem

# For an internal CA signed certificate:
openssl s_client -showcerts -verify 5 -connect <YOUR_CHEF_SERVER>:443 </dev/null 2>/dev/null | awk '/BEGIN/,/END/{if(/BEGIN/){a++}; certs[a]=(certs[a] "\n" $0)}; END {print certs[a]}' >> /opt/chefdk/embedded/ssl/certs/cacert.pem

export SSL_CERT_FILE=/opt/chefdk/embedded/ssl/certs/cacert.pem

The openssl command is included in chef-dk, so this can be done under windows also.  Change the path to c:\opscode\ instead of /opt/. To export the environment variable use set SSL_CERT_FILE=... (with /P to add it permanently to your environment) in your command.


Jenkinsfile: Pull image from registry endpoint A, push built image to registry endpoint B I have two endpoints for my registry (Nexus OSS):

Endpoint 10500 to pull images from.

And Endpoint 10501 to push images to.

(I can go into detail what the idea is and why we think Nexus requires this if need be. I might have Nexus configured wrongly after all. But I am currently thinking it's correct.)

Jenkinsfile has:

docker.withRegistry(myRegistry:10501, 'nexus_jenkins_user') {
    docker.build(imageToBuild.getName(), '-f ' + imageToBuild.tag + '.Dockerfile .').push();
}

Dockerfile has

FROM myRegistry:10500/myNamespace/core:latest
...

Jenkins execution reports the following error:

docker build -t myNamespace/php:php.7.2 -f php.7.2.Dockerfile .

Sending build context to Docker daemon   80.9kB

...

Step 1/7 : FROM myRegistry:10500/myNamespace/core:latest

Get https://myRegistry:10500/v2/myNamespace/core/manifests/latest: no basic auth credentials

script returned exit code 1

This seems obvious as the "withRegistry" directive above has the 10501 endpoint, so there would be no authentication for the 10500 endpoint here.

The challenge:
I need to be able to pull the core image (through 10500) but push the finished image to 10501.

What would my Jenkinsfile need to look like to achieve this?

This should theoretically be a common problem for companies utilizing Nexus OSS as they will have two endpoints as well. (Assuming I am correct about Nexus requiring two endpoints if a hosted repo and a proxy repo are being used).

edit
To provide some help for understanding: If I specify

FROM myRegistry:10501/myNamespace/core:latest

in my Dockerfile, authentication will work as "withRegistry" and the Dockerfile's FROM directive are set to the same endpoint.

But I want all pulling actions to go through 10500. If, for example, I would use an image from Docker Hub, 10500 would pull it from Hub, cache it and deliver it back to me, where 10501 would not. So 10500 needs to be used for all pulling.

Try to chain withRegistry like this:

docker.withRegistry('https://registry:10500', 'credentials-id') {
    docker.withRegistry('https://registry:10501', 'credentials-id') {
        docker.build(imageToBuild.getName(), '-f ' + imageToBuild.tag + '.Dockerfile .').push();
    }
}


Nginx-Ingress Controller are compatible with Network Security Policy or not? I have created one application where I use Nginx-Ingress Controller version 3.29.0. Without using network security policy I'm able to access my application. But, after applying network security policy, I'm unable to access my application.
I checked all logs and troubleshoots the whole application but there is nothing anything odd. all my services and nginx ingress controller are working fine. But still when I go to access my application it will show me error that, "To long to respond".
Can any one help me with this?
Is there anything like nginx-ingress controller is not compatible with network security policies or what?
and how can I solve this?
Nginx-Ingress Controller version - 3.29.0
Kubernetes version - 1.19.7
Thank you.

I don't think it is the compatibility, need to really see your configuration to be sure. But most likely, you block something wrongly. If you use the default network policy api networking.k8s.io/v1 notice that it only allows what is defined.


How to monitor a REST call response? My project is using customer.io to send emails in different flows.

Every now and then we have a bug, and one of those calls stops and email is not sent.

I want to get alerted when an email is not sent in the last 24 hours.
It would be great if the solution could give me alert on any spike but the mvp is 0 in the last 24 hours.

What is the best way to do it?

Currently, I am trying to use some process to poll customer.io REST API which provides exactly the information I need.

[{
    "campaign": "Hello",
    "sent": 0 // This should trigger an alert. sent is for last 24 hours
}, {
    "campaign": "World",
    "sent": 1 // should not trigger an alert
}]

Then to do some ETL on this, and feed some data source (maybe cloudwatch, haven't decided yet), and then hook grafana to give me alerts on any spikes.

I am new to this field and am wondering if I am going the right way and if there's a better/easier way to tackle the problem.

== Update - I found out about Grafana SimpleJSON plugin but it expects me to be the owner of the backend. I am going to write some lambdas and use gateway (basically implement an ETL) and use SimpleJSON plugin from grafana to get things going.

There's no generic answer to this, but a ton of options.  Usually the right choice for you would be dependent on what monitoring and alerting stack you're already using.

For instance, if you are using Graphite, Prometheus, rrdtool, or any other round-robin database to record a metric when you're sending out emails, those all have APIs that allow you to query them across a time range to see what the values were.  And depending on what alerting system you're using (Nagios, Icinga, Sensu, etc.) there is probably already a plugin that allows you to easily interact with that particular metrics database.

From another perspective, if you're logging when you send emails, then a log system like ELK will allow you to search and see if any of those log lines are found.  Again, the various alerting systems will have plugins to most common log systems.  (There are also individual systems designed purely for those tools, like ElastAlert with its flatline type for ELK.)

Thirdly, you could approach it the way that you are, by checking in with customer.io's API on a regular basis.  But no, there is not one standard tool for doing this, but rather one (or at least one!) per alerting system.

Finally, you might consider doing synthetic checks instead of or in addition to your real-user monitoring.  This means having an automated test that performs actions that you expect to result in sending an email, and then checking to see that an email gets sent.


Unable to send cloudwatch logs to loggly using a blueprint lambda - Invalid ciphertext I am trying to use the loggly Lambda blueprint to send cloudwatch logs to loggly. I am doing this because I have a bunch of lambdas that I need to have a single point of viewing logs. I've followed this guide and have run into an issue when I check the lambda logs after a test
Command used to do ciphertext
aws kms encrypt --key-id alias/logglyCustomerToken --plaintext "<my token>"

KMS key was created using symetric and default settings, can't delete key to try again or at least I don't know how to delete it and thinking I shouldn't change the code to match a new one
Lambda log message when failed test
INFO    InvalidCiphertextException: null
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'InvalidCiphertextException',
  time: 2020-07-08T02:48:55.536Z,
  requestId: 'cf0df165-5a59-4db0-beef-5d1ae3619c37',
  statusCode: 400,
  retryable: false,
  retryDelay: 79.27899980360486
}

Follow these steps to fix InvalidCiphertextException:

Open the Lambda function in AWS console.

Scroll down to the Environment Variables section & click Edit:

Delete the value of kmsEncryptedCustomerToken & replace it with the customer token you got from https://<your-company>.loggly.com/tokens. The token looks like this: 72cf6d64-256e-449d-aabd-49e1f422ed58.

Expand the Encryption Configuration section at the bottom.

Select Enable helpers for encryption in transit.

Click the Encrypt button that appears next to kmsEncryptedCustomerToken.

Select the logglyCustomerToken KMS key (CMK) & Encrypt.

Finally, Save the environment variables.

The Lambda function test should succeed now. If you face any other issues or need a complete end-to-end screenshot-guided tutorial for the entire process of sending CloudWatch Logs to Loggly, please see my blog post.


Coupling docker registry and source control What are the best practices (if any) for coupling a docker image registry with a SCM service (such as bitbucket)?

I know a docker registry can live in artifactory, but how can I assure that the two are as tightly coupled as possible?

For example I am worried that without due-diligence from developers the latest Docker image in the registry will not reflect the current state of the Dockerfile in SCM or vice-versa.  Also it is imperative that we can always trace back to the base Dockerfile (that we made) for any image in the registry.

I was able to get an answer over here forums.docker.com Credit goes to dmaze.

Set up some sort of automated build system (“continuous integration” in the current trendy term). Docker is sufficiently mainstream at this point that any of the cloud-based or locally-installed CI systems can do it.

In your Dockerfile, use a LABEL to record the source of the build. That probably includes the commit hash from distributed source control (git, Mercurial), the branch name if relevant, any release tags if present, and possibly details like the timestamp of the last commit. docker history and docker inspect should be able to show these.

When you docker push your images, push them at least twice, with the commit hash and with the branch name as the “version” part (quay.io/mycorp/imagename:123abc7, quay.io/mycorp/imagename:dmaze-test). If release tags are readily available, the CI system should push images with these tags too.

Make sure the Dockerfile is committed to source control, of course, and try to have a stable path to get any external dependencies that may be present.

Now you can go both ways: given an arbitrary commit, if your CI system built it, you can docker run the image it built; and if you have an image, you can find where exactly in source history it came from, and git checkout or hg up to that specific version, and docker build a near-identical copy of it yourself.


Workflow with docker in the development environment I have a couple of questions related to the usage of Docker in a development environment. I am going to propose three different scenarios of how I think Docker could be used in a development environment. Let's imagine that we are creating a REST API in Java and Spring Boot. For this, I will need a MySQL database.

The first scenario is to have a docker-compose for development with the MySQL container and a production docker-compose with MySQL and the Java application (jar) in another container. To develop I launch the docker-compose-dev.yml to start only the database. The application is launched and debugged using the IDE, for example, IntelliJ Idea. Any changes made to the code, the IDE will recognize and relaunch the application by applying the changes.
The second scenario is to have, for both the development and production environment, a docker-compose with the database and application containers. That way, every time I make a change in the code, I have to rebuild the image so that the changes are loaded in the image and the containers are launched again. This scenario may be the most typical and used for development with Docker, but it seems very slow due to the need to rebuild the image every time there is a change.
The third scenario consists of a mixture of the previous two. Two docker-compose. The development docker-compose contains both containers, but with mechanisms that allow a live reload of the application, mapping volumes and using, for example, Spring Dev Tools. In this way, the containers are launched and, in case of any change in the files, the application container will detect that there is a change and will be relaunched. For production, a docker-compose would be created simply with both containers, but without the functionality of live reload. This would be the ideal scenario, in my opinion, but I think it is very dependent on the technologies used since not all allow live reload.

The questions are as follows.

Which of these scenarios is the most typical when using Docker for development?
Is scenario 1 well raised? That is, dockerize only external services, such as databases, queues, etc. and perform the development and debugging of the application with the IDE without using Docker for it.

The doubts and the scenarios that I raise came up after I raised the problem that scenario 2 has. With each change in the code, having to rebuild the image and start the containers again is a significant waste of time. In short, a question would be: How to avoid this?

Thanks in advance for your time.

We use kubernetes / minikube rather than docker-compose, but its the same problem.
We generally go with option #1 as:

Development is 99% getting the app right.
You want to be able to iterate fast/freely with your app.
You want easy use of your debugger / IDE without inhibition.
It lets you avoid configuring / managing remote debuggers from within the docker container.

If you did this in the container, you'd have to worry about making sure you didn't actually put it in prod as it can be harmful; see below.

For most code changes, we actually just have GitLab-CI set up so that it test-builds/deploys the container/app on check in (on their own branch/space).  That way, most developers don't even need to think about the container part unless they're making mods to that area.  But, obviously you'd still want them being able to run the MySQL container locally and easily.

Remote Debugging Info

https://jaxenter.com/remote-debugging-java-applications-151466.html

  Warning: Don’t keep remote debugging JVM arguments ON always, as it has following downsides:

  Remote debugging mode disables several optimizations that JVM does to application to optimize the performance. All those optimizations will be lost.
  Remote debugging opens up a port. It’s a security risk, as anyone who can hit the server can initiate remote debugging


Accessing remote servers through a VPN tunnel from within docker container My laptop has an IPsec VPN tunnel set up to access a remote site (with my side being the VPN client).

As an experiment, I started up a simple docker container on my laptop using docker run --rm -it alpine:3.7 /bin/sh and from within the container tried to ping one of the remote machines on the other side of the tunnel.

I expected the ping to not work, but I was surprised that it actually did.

I expected that in order to be able to access one of the remote machines from within a docker container, I'd have to first set up the VPN connection from within the container.

If not a full-fledged VPN connection from within the container, I thought that I would at least need to set up some iptables rules on the host machine. (My laptop's private IP address is 192.168.10.1, that of the remote server is 10.2.5.100 and that of my docker container is 172.17.0.2.) But even this wasn't needed...

So my question is how is it that communicating with the remote machines through the tunnel just works without any additional setup?

p.s. in case it matters, my laptop is running Arch Linux and I'm using strongswan to set up the IPsec connection.

If you run docker run --rm -it alpine:3.7 /bin/sh, created docker container will use default bridge network.

You can see the properties of this network using the following command: docker network inspect bridge. You can see there to which host interface (usually docker0) the network is associated and also what is the subnet for this network (usually 172.17.0.0/16).

Additionally, if you run docker run --rm -it alpine:3.7 ip addr command, you should see the IP address of your container, which should be part of previously mentioned subnet.

Then, Docker by default creates NAT iptables rules, which allows the containers to communicate with outside world (as the container IP address is not known to the internet). You can see the rules using sudo iptables -t nat -nvL POSTROUTING command. It should look similar to:

Chain POSTROUTING (policy ACCEPT 144K packets, 15M bytes)
 pkts bytes target     prot opt in     out     source               destination
  392 75916 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0

Finally, for your container, there is no difference, if you try to reach remote site of your VPN or the internet, as the container only knows how to reach Docker network. You can see that using docker run --rm -it alpine:3.7 ip r command:

default via 172.17.0.1 dev eth0
172.17.0.0/16 dev eth0 scope link  src 172.17.0.2

So when traffic from the container goes via default route, it hits host routing table (I'm not 100% sure if this is right), it is being routed to your tunnel and then iptables NAT rule applies, changing the source IP of your traffic to IP of your end of the tunnel, so returning traffic will come back over the tunnel as well and then back to your container.


How to use script in a Gitlab bridge job? I have a job which needs to include another job and needs to append a value to a variable, such as:
nightlies:
  variables:
    BUILD_NAME: "nightly"

  script:
    - $BUILD_NAME="${BUILD_NAME}-$(date +"%Y%m%d")"

  only:
    - schedules

  trigger:
    include:
      - local: /.gitlab-ci/build.yml
    strategy: depend

However reading the documentation script and trigger cannot be used together and I'm unsure of how else to create the BUILD_NAME with a date. Is there a way to use both. Otherwise, is there a way I could remove the trigger in order to use script, which would still call that job?

One option is to inherit your environment variable from another job. You can create two different jobs, one to create the environment variable, and one to trigger the local CI pipeline.
Once all that is done, you should have something like this:
stages:
  - build
  - deploy

nightlies_env:
  stage: build
  variables:
    BUILD_NAME: "nightly"
  script:
    - BUILD_NAME="${BUILD_NAME}-$(date +"%Y%m%d")"
    - echo "BUILD_NAME=$BUILD_NAME" >> build.env
  artifacts:
    reports:
      dotenv: build.env

nightlies:
  stage: deploy
  variables:
    BUILD_NAME: $BUILD_NAME
  trigger:
    include: build.yml
    strategy: depend


How can a Sys. Admin skill up to work as a DevOps engineer? I am looking for recommendations on how to skill up to be a dedicated DevOps Engineer.

I have always wanted to work on Development related projects as I love coding but unfortunately I have never got the chance. I have previously worked on freelance projects using .Net and PHP and I am certified in Java SE but I have never used it.

My work experience incorporates being in system and application administration. Mostly I am a senior now in this field so a switch to Development is not preferred as I would have to start over as a junior. However I think DevOps will be perfect for me but I need to know what I need to study to fill the gaps in my knowledge to be a good dedicated DevOps engineer (preferably the part about writing automation and tools).

Please recommend what should I study, for example I heard Python and Perl and Ruby are very good for DevOps but what else should I start with and what is the mandatory skills that i need to cover first?

My current skill set:

Sys. Admin

Unix (HP-UX, Solaris) and Linux (Ubuntu, Red hat,
CentOS) Administration
Middleware Technologies administration
(WebLogic, Apache-Tomcat, WebSphere, MQ)
Bash/Shell Scripting
Web server configuration and administration
Excellent knowledge of common database
systems (SQL Server, Oracle, MySQL, etc.)
Web services and Integrations SOAP/REST/API
Oracle Business Intelligence Suite (OBIEE, DAC,
Informatica)
ITSM Remedy Development and Administration.
SAP Business Object (Analytics).
Splunk
Service Center –ITIL-enabled IT service
Management
Strong understanding of Web Application
technologies.

Developer

Expert Development skills in Microsoft Visual
Basic.
Excellent Development Skills in Java SE.
Microsoft .Net Development
MS SharePoint development.
Expert in MS Office with Expert knowledge in
Macros and Automation across all MS office
Applications.
Intermediate PHP development (CakePHP)

Have a look for DevOps Engineer roles and check out the list of technologies that they are seeking. Typically you'll need automation scripting skills (like Bash, Python or PowerShell), configuration management skills (like Chef, Puppet or Ansible) and some fundamental experience maintaining and configuring a Build/Release pipeline. For this, look to get experience with CI/CD tools like AppVeyor (.NET), Jenkins, Travis CI etc. You should also be familiar with Git and source control systems in general such as BitBucket or TFS. You would also benefit from becoming familiar with automated testing techniques.

If you haven't already, consider hosting some open source products in Github and setting up a pipeline for them that performs automated tests on commits as well as (if possible) publishing to a package management system. This is great experience as well as demonstrates your skills.

For a more general understanding of the philosophy of DevOps (as beware that it's a culture as well as a technology set) do as much reading as you can on the topic and start following (e.g via Twitter) the likes of Gene Kim, Jez Humble, and Patrick Debois. Also I recommend you read The DevOps Handbook which they authored. It is a very thorough explanation of what DevOps is as well as how organisations might adopt it.

You could also check out DevOps related Meet Ups and Conferences (such as Velocity, DevOpsDays and Config Management Camp). The opportunities will of course vary depending on where in the world you are.


If we are using containers do we still need to worry about configuration management at an infrastructure level? We are about to begin our journey with containers and I am wondering if we still need to utilize something like Chef, Ansible, Terraform, etc. to ensure that our infrastructure is properly maintained?

Since containers simply sit on that environment my assumption is that we will still need to properly maintain our configuration idempotence but how is this going to differ with us using containers?

Welcome to containers, a very exciting direction indeed.

Software configuration management tools such as the ones you mention do cater for containers in many ways and if you're already familiar with Terraform then you'll probably already know this.

Kubernetes for instance can manage deployments and replication sets (of containers; 'pods') and nodes much like Puppet might manage virtual machines and micro-services.

I would say I personally use Puppet less in the container world but believe configuration management is still necessary in a container infrastructure though you may change the way you approach it with regard to the native tools that come along with containers as opposed to Puppet perhaps.

A Dockerfile is fairly self explanatory to satisfy the infrastructure as code perspective that Puppet delivers; and if you then get into Kubernetes clusters and nodes the same could be said about YAML files too.

Containers are bundles of repeatable infrastructure. I find this stack overflow answer supports my answer too; it certainly helped me in the past.

https://stackoverflow.com/a/41472271/1524645


How to recreate GCP Preemptible VMs behind LB before they expire? I created preemptible instances in a managed instance group behind LB, I specified the preemptible option in the instance template.

Because GCP Compute Engine always terminates preemptible instances after they run for 24 hours, I need a way to launch new preemptible VMs before they goes down.

Is GCP Compute Engine launch new preemptible VMs before the current ones goes down when target size specified by the autoscaler ?

Maybe need to replace the Preemptible VMs several hours before they going down,

I am not sure what is the best way to do it, maybe with termination notices.

Basically what I need is always to keep on Preemptible VMs behind the LB.

What is the best way to implement this?

Strictly you can't do this with a single preemptable virtual machine as you only get a 30 second grace period. While the majority of calls to the Google Compute Engine API would create a new device within 30 seconds, there are cases when this would fail due to network latency or temporary outage.

What you can do is create an Instance Group of Preemptible Virtual Machines with a minimum of two instances, the probability of GCE killing both at precisely the same time is quite low, and once GCE destroys one a new one will be created to bring the IG instance count back up to two.


Nginx healthcheck does not stop I have the following healthcheck condition for a nginx container inside my docker-compose.yml file:
healthcheck:
  test: ["CMD", "curl", "-f", "http://front-service:80"]
  interval: 15s
  timeout: 5s
  retries: 5

The problem is that I see in the logs continuous calls to http://front-service:80 even after nginx is running fine.
front-service    | 172.18.0.4 - - [02/Mar/2021:09:58:23 -0300] "GET / HTTP/1.1" 200 2611 "-" "curl/7.64.0" "-"
front-service    | 172.18.0.4 - - [02/Mar/2021:09:58:38 -0300] "GET / HTTP/1.1" 200 2611 "-" "curl/7.64.0" "-"

Wouldn't the correct behaviour be that if the check succeeds, it stops pinging my frontend endpoint? I don't want it to keep pinging after it finds out that the server is alive.

If you see the logs, there's exactly 15s between the calls. Which is what you have configured in you docker compose file.
If you don't want it to keep pinging after it finds out that the server is alive, you have two options (which is not recommended to do):

Don't use healthcheck, just add your logic in the command part
Setup a higher interval for your healthcheck

Also, I have questions for you:

Why do you want the healthcheck to be done only once?
What if you server goes down, don't you want to restart it?

My recommendation is to keep this config as is because it does the work of checking for your service availability and will restart as needed and quickly.


Why do many pods fail to communicate with the master I have lots of unexplained errors in kubernetes. Often I see errors in logs of pods with the following:

Get https://10.152.183.1:443/api/v1/namespaces/{REDACTED}: x509: certificate is valid for 192.168.1.156, not 10.152.183.1}

I am using Canonical's distribution of Kubernetes deployed with Juju on a MAAS cluster.

Try to remove the easyrsa relation with kubernetes-master in juju.

juju remove-relation easyrsa kubernetes-master

Recreate it afterwards:

juju add-relation easyrsa kubernetes-master

The certificate seems to be recreated for the master, which should now include it's clusterip (10.152.183.1) address as well in the SAN record of the certificate.


Multi jobs over local (IntelliJ) and Jenkins I have to build a way for my tests to work with multi jobs over any platform using IntelliJ and Jenkins

I have heard that I can do this with JUnit 5. I have not been able to find any references about this.

Can someone with experience in setting up multi jobs in Jenkins point me to some good references?

You can check out this MultiJob plugin for Jenkins. Or if its the same job you want to run in different configuration you can make use of Multi-configuration job type in Jenkins.

I have tried searching for JUnit5 supporting multi jobs scenario as specified didn't find any decent information on the same.


Are using TestContainers safe for integration tests? I have a discussion with our DevOps about using Testcontainers for integration tests. It's a tool to bring up throw-away containers for testing. The testcontainers need a environment variable DOCKER_HOST=tcp://127.0.0.1:2375 to communicate with Docker.

He refuses to set this environment variable explaining to me that he don't want to application communicate with Docker for security reasons.

I've make a point that tool is designed to interact with Docker when running tests.

So here is my questions:

Are there security considerations in this regard?
What more convincing arguments can I bring?

I think the question about Docker security in general was already well answered in other places, for example here: https://stackoverflow.com/questions/40844197/what-is-the-docker-security-risk-of-var-run-docker-sock

In your particular instance, I see the following points:

The tests should run on some testing server, possibly even in a VM that is destroyed/reset after each test run, so possible accidents/incidents should not affect other systems
For an attacker to use this setup, they would probably (depending on your tests) need to be able to modify your test code and if they are able to do so, you will already have much bigger problems than privileged Docker access


How to automatically update Docker containers in development? What is the best way to use Docker in development so that a container is automatically updated each time you make a change to the codebase?

An approach I use in both prod and dev is to run a webhook server ( this one is excellent https://github.com/adnanh/webhook ) which is subscribed to listen to the github.com  git push events which are published by github

... then when someone does a push to github.com the webhook launches a rebuild script to issue git pull then recomple the code and issue docker build xxx and docker push yyy  and docker-compose -f my-file-docker-compose.yaml up   ...  this all happens on auto pilot ...  where I have a running webhook server on a vps box for a given git branch to rebuild its toy domain

this took time to craft however its a life saver


helm - cannot get resource "namespaces" in API group I installed Kubernetes, helm on the Ubuntu 16.04.6 Box, I want to deploy kubernetes-dashboard using helm but getting below error while installing it

$ helm install stable/kubernetes-dashboard --name dashboard-demo
Error: release dashboard-demo failed: namespaces "default" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "default"

$ helm list
Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "configmaps" in API group "" in the namespace "kube-system"

Tools Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

$ helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS             RESTARTS   AGE
kube-system   calico-kube-controllers-58dc446999-zrwz6   1/1     Running            0          4h41m
kube-system   calico-node-wjczq                          1/1     Running            0          4h41m
kube-system   coredns-fb8b8dccf-fp9gx                    1/1     Running            0          4h41m
kube-system   coredns-fb8b8dccf-q574t                    1/1     Running            0          4h41m
kube-system   etcd-ubuntu1                               1/1     Running            1          4h40m
kube-system   kube-apiserver-ubuntu1                     1/1     Running            1          4h40m
kube-system   kube-controller-manager-ubuntu1            1/1     Running            2          4h40m
kube-system   kube-proxy-ztmms                           1/1     Running            1          4h41m
kube-system   kube-scheduler-ubuntu1                     1/1     Running            2          4h41m
kube-system   tiller-deploy-864c6f45bb-2585b             1/1     Running            0          6m55s

helm updated to latest version.

I followed this : https://stackoverflow.com/questions/46672523/helm-list-cannot-list-configmaps-in-the-namespace-kube-system but still facing same issue, how can we fixed it ?

Check https://github.com/helm/helm/issues/3130, this might help.
I followed the instruction in the post:

kubectl --namespace kube-system create serviceaccount tiller

kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller

helm init --service-account tiller --upgrade

This works for me.


Jenkins Pipeline - Clean before checkout - git clean -fdx "failed to remove: Directory not empty" I have my Jenkins build configured to "Clean before Checkout" for its git source:

Job Config > Source > git > Behaviours > Clean before checkout

This results in

Resetting working tree

git reset --hard # timeout=10
git clean -fdx # timeout=10

Before the job begins.

This errors out with

stderr: warning: failed to remove web/static/node_modules/.../fonts/google-fonts: Directory not empty

If I re-run the job (simply replay), I get the same error.

If I re-run it again, git clean works fine and the job finishes with success in the end.

If I start it again it errors out 2 times again until it works at the third time. This is always the case.

What might be going on here?

We didn't find the cause, neither a real solution.

There is a workaround though:

Put a pipeline stage with rm -rf * before your scm checkout stage. This solves the problem but I will change the accepted answer to whoever posts the real reason and fix for the problem.

Sidenote:

Keep in mind that you will erase anything in your Jenkins workspace doing that. So be certain that you have everything you need in Git so it comes back with the checkout or get it back some other way.

If you rely on files in your Jenkins workspace from previous runs then this method is not for you!

(Although you might want to think about adapting your Job so files from previous runs are not needed anymore. Every run should contain the files it needs by itself and not rely on other runs.)


How to revert jenkins Safe restart? Jenkins was having issues and I safe rebooted it in order to solve the issue but the issue got solved the reboot. Some jobs have ours to run so is there way to cancel the reboot?

I assume you talk about jenkins cli?
There is no command to abort the safeRestart.
Use quiteDown, cancelQuietDown instead and safeRestart only if needed.

safeRestart: Puts Jenkins into the
quiet mode, wait for existing builds to be completed, and then restart
Jenkins
quietDown: Put Jenkins in a Quiet mode, in preparation for a restart. In that mode Jenkins don’t start any build
cancelQuietDown: Cancel the effect of the “quiet-down” command


Where is Puppet's enviroments directory on Amazon Linux2? I am using Puppet version 3.8.7 on Amazon Linux2, trying to find the directory containing the Puppet environments.

I tried to find it with the find command looking for files named "production" or defaults, following the names used in a tutorial using ubuntu, but there is no production or default file or directory on the Amazon Linux as there is on ubuntu version.

The distribution of Puppet I downloaded was

http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm

You tried look on /opt or /etc probably the puppetlabs folder must be in one of these directories.
Amazon Linux it's based on CentOS/RedHat family.


Have Jenkins trigger GitlabCI Assuming a user initiates a Jenkins job, passing some input parameters.

Is there a way, at some point to have the Jenkins job trigger a gitlab-ci, and also pass to it (i.e. the gitlab pipeline) some input parameters which the latter will use as environment variables?

Note: I didn't use this yet, the answer is based solely on documentation.

A GitlabCI pipeline can be triggered via API, see Triggering pipelines through the API.

  Triggers can be used to force a pipeline rerun of a specific ref (branch or tag) with an API call.

You would first need to create a trigger for that pipeline.

Then you can activate that trigger (from your jenkins job in this case):

  To trigger a job you need to send a POST request to GitLab's API
  endpoint:

POST /projects/:id/trigger/pipeline

  The required parameters are the trigger's token and the Git
  ref on which the trigger will be performed. Valid refs are the
  branch and the tag.

Such triggers support variable passing:

  You can pass any number of arbitrary variables in the trigger API call
  and they will be available in GitLab CI so that they can be used in
  your .gitlab-ci.yml file. The parameter is of the form:

variables[key]=value

You can use the variables inside .gitlab-ci.yml like this:

upload_package:
  stage: package
  script:
  - if [ -n "${UPLOAD_TO_S3}" ]; then make upload; fi

And

  You can then trigger a rebuild while you pass the UPLOAD_TO_S3
  variable and the script of the upload_package job will run:

curl --request POST \
  --form token=TOKEN \
  --form ref=master \
  --form "variables[UPLOAD_TO_S3]=true" \
  https://gitlab.example.com/api/v4/projects/9/trigger/pipeline

This kind of POST requests is what you would to do in your jenkins job. But take a closer look at the triggers, you might find a way to do what you want without jenkins.


Original Helm chart gone - how can I find / get it from the cluster? I'm connected to a cluster which has a release, from a chart, but it was installed by someone else who I think used a tweaked, locally-stored chart.

I don't have access to their original chart/repo. Is there a way I can retrieve it from the cluster, so I can tweak/amend, and re-install it?

Or, can I find the URL of the original repo that was used? The chart name is wordpress-0.6.0, but that's of no use, since it's the same as the "official"/original one.

You can use helm get manifest <RELEASE_NAME> to fetch the kubernets manifests and you can use helm get values <RELEASE_NAME> to get the values.yaml from the cluster. But I don't think there is any way to get the chart files back. Sorry.

PS: Please replace <RELEASE_NAME> with your release name.


kubernetes How to test if an operator has already been applied How can I check if an operator has not changed, and then not apply it?

usually, I would apply an operator like this

kubectl apply -f openebs.yml

In an automated script, I would like to test if Kubernetes is running this operator already - and only apply it if it has changed or even stop if there are changes.

How can I achieve this?

Not sure why this is required since Kubernetes is declarative and will not do anything is nothing has changed. Nonetheless, you can run it with the --server-dry-run=true flag. What attributes are going to be changed in the resource will not be mentioned but it does mention which resource(s) will be changed in the apply. You will need to parse the output message to identify if something changed or not since the exit code is always 1.
Sample outputs:
$ kubectl apply --server-dry-run=true -f ./configmap.yaml # First run
secret/my-conf created
$ kubectl apply --server-dry-run=true -f ./configmap.yaml # Re-run without modifications
secret/my-conf unchanged
$ kubectl apply --server-dry-run=true -f ./configmap.yaml # Re-run with modifications
secret/my-conf configured


How to generate a Maven.pom file (not pom.xml)? This seems not so easy to google because results about the normal pom.xml are overwhelming.

So, I just have learned there are also Maven artefact related files with .pom extension. This seems to be some special thing about how repositories work - a meta-file for an artefact?

For example, if you check folder contents for an artefact in your local repository, you will find these files in any artefact folder I guess, just like in the path below.

\.m2\repository\org\json\json\20090211\json-20090211.pom

Contents:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <!-- this POM is released under an Apache 2.0 license-->
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.json</groupId>
  <artifactId>json</artifactId>
  <version>20090211</version>
  <name>JSON (JavaScript Object Notation)</name>
  <description>
    JSON (JavaScript Object Notation) is a lightweight data-interchange format.
    It is easy for humans to read and write. It is easy for machines to parse and generate.
    It is based on a subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition
    - December 1999. JSON is a text format that is completely language independent but uses
    conventions that are familiar to programmers of the C-family of languages, including C, C++, C#,
    Java, JavaScript, Perl, Python, and many others.
    These properties make JSON an ideal data-interchange language.
  </description>
  <url>http://www.json.org/java/index.html</url>
  <licenses>
    <license>
      <name>provided without support or warranty</name>
      <url>http://www.json.org/license.html</url>
    </license>
  </licenses>

  <organization>
    <name>JSON</name>
    <url>http://json.org/</url>
  </organization>

  <dependencies/>
</project>

Question: What is exactly this file and is it possible to generate this file as part of a Maven build? Or do we have to create these files manually as part of the codebase ?

(In the context of what I am doing, I use generic Artifactory deployment plugin for Bamboo to deploy artefacts there but this .pom file is not being created automatically).

UPD I have solved my problem by using a copy of the pom.xml as this pom file and it works but I consider this can not be the correct way.

Answer from here:

  Every jar needs to have a pom file describing it, you can just add
  something simple like this:

Every jar needs to have a pom file describing it, you can just add something simple like this:

http://maven.apache.org/maven-v4_0_0.xsd">
    4.0.0
    aGroup
    aArtifactId
    aVersion
    jar
    a Name


Getting data from crashed docker container I  am having docker container with Ubuntu installed  which was running with Nvidia docker
But now I connected that HDD in a different computer. Now After typing
 following command in newly assembled system

  sudo docker start container

I am getting error something like
Unable to create  /etc/nvidiactl

It seems like it needs Nvidia GPU to run that container which I don't have. So how can I get the data which is there in the container if i am unable to start it?

So what you can do is you list all of your containers with command:

docker container ls -a

And the second field is an image name, you can run this command to get all the read-only layers with their respective directories to search for the file in:

docker image inspect <image_id> | jq '.[0].GraphDriver.Data.LowerDir' -r | tr ":" "\n"

If the file was something written inside the container replace Lower with Upper.

In case you are on Mac the /var/lib/docker is inside the virtual machine and so you have to first enter that before you can get to those directories described in the command above. You can do it using nsenter like this:

docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh


Docker swarm replica stays 0/1 and service stays in status new I have a problem with my docker swarm. I created a new service with the following command:

docker-compose config | docker stack deploy -c - serviceNameX --with-registry-auth

The docker-compose file looks like that:

version: '3.1'
services:
  serviceNameX:
    image: ${DOCKER_REGISTRY}imageNameX:${VERSION}
    deploy:
      resources:
        limits:
          memory: 2gb
    env_file:
      - ../common.env
      - .env

networks:
  default:
    external:
      name: ${DOCKER_NETWORK}

fyi: docker-compose config is used because docker stack deploy can't expand environment variables. At least to my knowledge.

docker service ps serviceNameX

ID                  NAME                              IMAGE                                                         NODE                DESIRED STATE       CURRENT STATE       ERROR               PORTS
ify7vhl0l2m8        serviceNameX.1   nexus.local/imageNameX:1                                                                           Running             New 2 hours ago

docker service ls

afnjo97f6nuf        serviceNameX                      replicated          0/1                 nexus.local/imageNameX:1

All nodes are Ready and Active

There is no indication or information of an error. Usually docker service ps shows if an error occurs. Is there another useful log?

Try do pull your image manually first.
docker pull ${DOCKER_REGISTRY}imageNameX:${VERSION}

And then re-run your compose-file
It may be a bug using a private registry, or something simple as you don't have enough disk space to pull the image.


AWS Task definition memory for Ubuntu docker image We are using alpine image on AWS ecs which work fine on just 100mb memory in task definition.
Now we need ubuntu image which size is more then 180mb while alpine is just 5mb.

I want to know that we need to increase memory according to image?
or Docker image run completely in memory or just current processes in memory what if I specify memory less then docker image size in task definition?

AWS recommend 300 to 500mb but in development I want to save memory.

Thanks

There are two important parameters that you can use wisely to deal with AWS ECS task definition.

memory:

  The hard limit (in MiB) of memory to present to the container. If your
  container attempts to exceed the memory specified here, the container
  is killed.

memoryReservation:

  Docker attempts to keep the container memory to this soft limit;
  however, your container can consume more memory when it needs to.

I recommend you to go with memoryReservation. So your task definition will look like this.

{
  "family": "my-alpine",
  "networkMode": "bridge",
  "containerDefinitions": [
    {
      "image": "alpine:latest",
      "name": "my-alpine",
      "memoryReservation": 100,
      ...
      ...
    }
  ]
}


DevOps Newbie - How to automate windows infrastructure deployment? DevOps newbie here, so bear with me.
I have a project where infrastructure deployment and application deployment is done using powershell and it is tied to VMWare vCenter.

For every client deployment, a deployment VM is created manually on their vCenter >> everything (application binaries and automation scripts) is copied manually to this VM >> DevOps run powershell script on this VM to create other VMs, configure them, and deploy application binaries.

What you would recommend to improve this process ?

The first thing I can recommend for you is to remove all of the manual steps in your deployment pipeline. DevOps is all about automating as much as possible to allow for fast, frequent, and safe deployment of your infrastructure.

Second, if you haven't already, I would recommend using a build and deployment tool (Jenkins, Bamboo, GitLabCI, etc.) to compile your binaries and execute your automated deployment/configuration scripts. Having a central location for your deployments allows you to be able to scale more effectively, avoid repetition, and monitor your deployments more accurately. You should also store ALL of your scripts in some form of revision control ASAP (if they are not already).

Finally, I recommend you read up more on DevOps best practices and how it can affect your organization. In order for DevOps to be truly effective, your entire development team and company must adopt best practices and be 100% on board with a shift in how you manage your deployments.

Some reading I can recommend are:

The Phoenix Project by Gene Kim, Kevin Behr, and George Spafford

A telling of a fictional, but very realistic story of DevOps transforming a company and the opticals that the team faces to get there.

The DevOps Handbook by Gene Kim, Patrick Debois, John Willis, Jez Humble, and John Allspaw

Outlines the steps/stages to transform your company into a DevOps organization along with an assortment of real life DevOps transforming case studies.

Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim

A plethora of empirical data and analysis on how DevOps has improved organizations both within IT and the entire company.

Continuous Integration by Paul Duvall, Steve Matayas, and Andrew Glover

Lays out the steps to take to move your organization to be able to deploy to the highest standards possible.


Docker compose host variable in extra_hosts not working I have a docker compose file with .env file, for example the .env file has this HOSTNAME=a1.test.com, and a docker-compose file has this:

...
extra_hosts:
  - "${HOSTNAME}:1.1.1.1"
  - "test.com:1.1.1.1"

When I run the container the /etc/hosts file content is:

1.1.1.1
1.1.1.1    test.com

So the docker compose not putting the variable data during the run process.

I tried this hostname: "${HOSTNAME}" and it is working fine the host name of the container is correct a1.test.com.

Any idea how can achieve what I want?

I've tried your case, and it works fine on my side:

version: '2'
  services:
    test:
      image: ubuntu:16.04
      command: sleep 9999
    extra_hosts:
      - "${HOSTNAME}:1.1.1.1"
      - "test.com:1.1.1.1"

and here is what I get in the container

cat /etc/hosts
...
1.1.1.1 a1.test.com
1.1.1.1 test.com


Are there dialects of YAML or just differences in parsing it? We use YAML in several frameworks to define types and structures for our deployments. I read an article about why some people don't like YAML and they stated that the parsing of YAML is very different depending on the interpreter.

Can it be said that these interpreters define their own dialects of YAML or would that not be the case?

The background is that we would like to verify and validate YAML in our templates.

There is only one YAML spec. That would lead me to say "There are no YAML dialects". There are certainly differences in implementation, perhaps due to what the article claims as the spec's ambiguity:

  YAML specification is so ambiguous,

It's a matter of opinion, I believe, but there is no doubting that various language-specific parsers of the same data result in different output. So, to answer the question(s):

no there are no YAML dialects (since there is one spec)
yes the spec provides much leeway for implementation and thus there are differences in how it is parsed.


How to use ecs service dicovery I'm trying to use amazon service discovery to register a Redis server in order to be reachable by other ECS services in the same cluster.

I'm using EC2 to deploy my services not Fargate, so service discovery create a simple SRV DNS record, that's mean my others services must request R53 service to get a real "A" record to have access to Redis service.

My question for DevOps/AWS engineer is: what's the correct way to do so? A request to R53 each time absolutely not a good idea.

My solution is in comments.

The goal can be achieved via several solutions. We can use service with awsvpc as network mode, so your service can get a real "A" DNS record and ENI interface, you must pay attention with this solution, ec2 instances has a limited number of ENI that can be attached, for a micro or small I think max 3.

Other solution more flexible is to use an internal NLB or custom proxy like HAproxy as ambassador, you run the proxy as service with awsvpc so he can get his "A" DNS record eg proxy.prod, all other services like Redis, Mongo, Elasticsearch ... can be configured as service with bridge networking mode so they will get an SRV record if configured with service discovery.

Finally you have to configure your proxy to redirect all traffic to desired service using port forwarding. HAproxy can resolve "SRV" records so you don't have to deal with that HAproxy will do it for you, you must just configure your app to use proxy.prod + port, for each service so:

Redis proxy.prod:6379
Mongo proxy.prod:27017
And so on...

Using this technique you can do a lot of amazing things, no matter where your proxy can lives, it can run as container in the same node with your apps or on a Fargate or in another node ...


How to reconcile Infrastructure as Code and not storing env config in code? When developing a Web App, and trying to adhere to what is generally considered best practice, e.g. The 12 factor app methodology etc. One key concept is to keep configuration and sensitive data out of your source code - access keys, connection strings etc. Using something like .env files to set different connection strings for a dev, test and production environment.

However, another key point (not sure is is mentioned in the 12 factor app), seems to be Infrastructure as Code. I completely see the benefit of this, however, I don't see how it can be reconciled with not storing/hard coding sensative data. Surely to entirely automate the provisioning of a new environment, a series of env variables has to be specified - in code...

How can these two seemingly contradictory pieces of standard practices be reconciled?

My current approach is something like this:

Hosted Git Repo with however many dev branches and a master branch
When dev branches are merged with master branch, the CI build is triggered, this builds the project and runs unit and integration tests.
The git repo also has a production branch, when master is merged into production, the same build runs, but if it is successful, a CD release is created. (In my case this is a deployment to an Azure App Service)

So my three environments are the local dev machines (before a PR into the master branch in the hosted repo), the CI build server and the production app service. At the moment, locally we have a .env file which isn't checked into the git repo. On the CI server, the NODE_ENV value is set to test, so the code uses hard-coded unit test values (nothing sensitive). Then in production we manually set the connection strings etc. in the app service settings. This, I feel, is potentially an issue.

So possibly I could add an infrastructure folder to the repo, with Terraform code in it, then as part of the release process, terraform plan could see if there is anything that needs changing between the code's infrastrucutre and the real world infrastructure (and if so, execute it). But where do I store the production env variables?

There are, as always, a few ways to solve this.

You can use a central source to keep secrets that each server reads from ala Hashicorp Vault. While popular this is not my preferred approach as its rather complex. There are quite a few key value stores that can provide similar functionality such as AWS Parameter Store.

You can manually put data in these stores and/or store it encrypted in git. Using tools such as kops (AWS only) or ejson (provider agnostic) you can have the secrets in the same repo as the app. Then have the app either fetch all the secrets at deploy time or just the decryption key so it can load the encrypted file.


How should I deploy my projects using Docker, Git and VPS? I have a VPS and am able to connect trough ssh using Terminal.
Every project I have are dockerized and have github repositories.
Since I am new to deployments so far my deployment process is manual:

$ ssh root@xxx.xxx.xx.xxx
$ cd /root/www/project_dir
$ git pull
$ docker-compose up --build -d --force-recreate

I would like to know how should I correctly deploy dockerized github applications on VPS? Is there a better practice or script, maybe a tool?

Here is a Webhook rule to catch Github release events and run a custom bash script. It is run by adnanh/webhook which is a small go app that can run your deployment script. You then configure GitHub to send release events as per this screenshot:

You can then use the github hub command line tool to create releases with a name that matches the regex in the hook rule e.g., v1.2.3. That will fire the release webhook. That can be caught by your deploy script running in a container on your server. Your deploy script can take the release number parsed from the release event payload. It can then git pull the git tag v1.2.3 launch your code. Note that you don’t have to use the hub cli a release event will also be fired if you push a tag to your repo using regular git. We find it helpful to use the hub tool that creates the tag and a github release. You can attach automatically generated release notes using github_changelog_generator see this example script.

Using similar webhooks is how OCD can drive kubernetes from configuration in GitHub. We have OCD deploy all our apps on AWS based on configuration and tags in GitHub. You can use the same techniques to drive your small deployment scripts quite easily. All you need to do is create docker container with adnanh/webhook, the json config and your deploy script. Here is a where we install webhook into a container and here is where we copy our rules. As it is a go app it doesn’t need a lot of memory to be able to fully automate your software based on git webhook events.