How to get Github username variable in Azure DevOps build pipeline? I have an Azure DevOps pipeline build configuration which builds automatically with the master branch. I want to access the GitHub user or username who committed last.

If I access variables $(Build.QueuedBy) and $(Build.RequestedFor) in a build that I triggered manually from the portal, I get the outputs as Sith and Sith respectively.

But when I pushed code to master and triggered automatically with CI, I get outputs for the variables as Microsoft.VisualStudio.Services.TFS and Microsoft.VisualStudio.Services.TFS respectively.

I can access all other variables like $(Build.DefinitionName) and $(Build.BuildNumber)
 in all the cases without any issues.

How to get Github username variable as Sith instead of Microsoft.VisualStudio.Services.TFS?

This is not supported by Azure DevOps pipelines yet. Microsoft will provide it soon. Will update this post once they made it available.

Pretty straight forward way to get last commit user in Bash:
git log -1 --pretty=format:'%an'


Application configuration of a complex enterprise system For a complex, distributed Enterprise system running on premise, I'll be driving an initiative to harness application configuration management. Today we have software delivered by almost 20 various scrum teams with only some coordination, having central and embedded software in Java, .NET, C++ and SQL worlds (MSSQL+Oracle). Creating a consistent system configuration addressing project specific functional requirements every time becomes a painful project.

What I would like to introduce is a type of configuration meta model starting from the business layer (domain of customers, product managers, etc) going down to individual parameters (domain of developers an system engineers). I imagine something like a web car configurator which in the end generates a consistent configuration. The whole complexity of relations between functionalities (hundreds)  and parameters (tens of thousands) would become explicitly maintained in such a tool.

Now, are there any off the shelve tools/frameworks supporting something like this? Gut feeling tells me that adapting any existing solution to a very specific needs of a large enterprise system might actually be more complex than building it by ourselves.

I highly recommend Consul for configuration management as it is a high reliability, distributed system that provides quite a few and easy to use features with a simple Rest API. Also the community edition is free :), although for a large enterprises there is an enterprise offering.

I am a particular fan of the Key Value store, because it allows me to have separate configurations for my micro-services and back-end services (written in C#/.Net Core and Java respectively) in all three environments (Dev, Beta, Prod).

Whenever I add a new service or application, the configuration for each environment is added to the Key Value store and a simple config client downloads the configuration during the application's bootstrap/loading routines.

You can read a bit more here but in a nutshell:

  Consul has multiple components, but as a whole, it is a tool for
  discovering and configuring services in your infrastructure. It
  provides several key features:

  Service Discovery: Clients of Consul can provide a service, such as
  api or mysql, and other clients can use Consul to discover providers
  of a given service. Using either DNS or HTTP, applications can easily
  find the services they depend upon.

  Health Checking: Consul clients can provide any number of health
  checks, either associated with a given service ("is the webserver
  returning 200 OK"), or with the local node ("is memory utilization
  below 90%"). This information can be used by an operator to monitor
  cluster health, and it is used by the service discovery components to
  route traffic away from unhealthy hosts.

  KV Store: Applications can make use of Consul's hierarchical key/value
  store for any number of purposes, including dynamic configuration,
  feature flagging, coordination, leader election, and more. The simple
  HTTP API makes it easy to use.

  Multi Datacenter: Consul supports multiple datacenters out of the box.
  This means users of Consul do not have to worry about building
  additional layers of abstraction to grow to multiple regions.

  Consul is designed to be friendly to both the DevOps community and
  application developers, making it perfect for modern, elastic
  infrastructures.


How the use Docker exec to update code into a running container I am using docker-compose to manage Docker. Now suppose I want to update a source code in one of the Docker containers.

Can I do it using the exec but then how to copy the code from host to running container ?

Or is there any better solution to do the above task ?

It's a bad practice.
You need to use Continuous Integration rebuild docker image (see Dockerfile) every time when source code changed.


Kubernetes multi cloud I'm exploring doing kubernetes on a multicloud setup the configuration I want is a master connected to nodes in multiple cloud platforms. In my limited research I haven't found many tools to make this happen,   from what I can tell rancher seems like the best option, but I don't think this fits within the traditional usecase. My question is does this sound like something rancher was made for? Honestly it seems more of a kubeadm usecase but the caveat is I want to use multiple managed solutions where multiple masters all share a common node pool.

(Rancher employee) Kubernetes itself is really not meant for that use-case.  A cluster is generally a set of machines in the same provider with close proximity (latency) to each other.

Etcd (and therefore API/CLI/UI/scheduling/everything) performance depends heavily on the (worst-case) latency between all members.
Only one cloud-provider integration (storage providers, L4 load balancers) can be configured and most assume nodes are all in the same "region" or similar concept.
Commonly used network plugins assume adjacency and/or provide no encryption or authentication suitable for communicating across untrusted networks/the internet.
Communication between pods/services within the cluster has no consideration for where the "nearest" place to reach a pod for that service is, so latency from a pod to another service can be large (however far apart the two worst nodes are) and unpredictable (sometimes close, sometimes far).  (There is an option for nodePorts to always go to a local pod, but if there is no local pod it will just get dropped).

You can make a "custom" cluster and add nodes from wherever and do what you're asking in Rancher, but you're not going to have a good time (no matter what product is building the cluster).


Helm Installation fails with shared config maps I have a couple of deployments in Kubernetes packaged using Helm. Both the deployments are different but they share the same config map.
When performing installation via Helm the first deployment is successful but the second deployment fails with the below error.

Error: rendered manifests contain a resource that already exists. Unable to continue with install: ConfigMap "appconfig" in namespace
"testnamespace" exists and cannot be imported into the current
release: invalid ownership metadata; annotation validation error: key
"meta.helm.sh/release-name" must equal "app2": current value is
"app2"

Couple of possible solutions:

Control the configmap from a third chart and add the third chart as a dependency in both the charts. This will duplicate the secret and can cause a skew in the values being used both the charts since if they have independent release cycles
Create the configmap in one namespace and refer to it using an external reference. You can see several examples of such at https://github.com/helm/charts/search?q=existingSecret&type=code

I personally prefer option 1 since there is explicit dependency specification and gives the ability to deploy the charts individually


Docker security risks passing secrets over environment variables I'm new to Docker and want to put my Django web application into containers.

For the last couple days I'm struggling with passing credentials safe to containers.

Online there is always the hint that secrets should not be passed as environment variable because they can be retrieved via inspect or from the images. My hope is that it can be neglected for my setup.

The setup

I provide the credentials via .env file of docker compose. This file is not on version control.
The containers are running as non-elevanted users (a different special user).
Docker command on the host can be used as root only.
I'm the only user of the host server (Ubuntu 18.04).
I assume breaking into the host is much harder (I'm using SSH public key authentication with non-root user only) than breaking into container of the Django app.
I don't want to use kubernetes or swarm.

I'm newbie to Docker and not a security expert. Do I have overlooked something? Why should I care about the docker inspect or images in this setup?

Thank you in advance

ENV variables are a good way of passing secrets to a container. What you don't want to do is bake the secrets directly in the image. As you have rightly done, do not store secrets in your source control repo as well.

Besides ENV variables, you could also consider using an external secrets repository, such as Hashicorp Vault, or possibly even AWS secrets. See also vault's github page. Your container would still need a token/credentials required to Authenticate/Authorize with the key store, but if required, these can be made to expire/rotate.

So you are on the right path. Some additional reading is worthwhile at this point:

https://medium.com/@mccode/dont-embed-configuration-or-secrets-in-docker-images-7b2e0f916fdd
https://techbeacon.com/how-keep-your-container-secrets-secure


Heroku Slack integration without using Github Heroku has this great Slack chatbot (chatops) that lets you send messages if a deployment pipeline succeeded or failed. Unfortunately, while setting it up and logging in to Heroku, it asks for your github account and will not let you chose a different DVCS provider.

Since I am using Bitbucket, this leaves me without a solution for pipeline notifications. Has anyone solved this in the past?

ChatOps uses GitHub status checks, so it's decidedly GitHub specific and won't work with another provider.

However, if you're just after notifications, what about Heroku's App webhooks? They have an 'entity' for new app releases (api:release) which might partially help you... in that you can set up a notification for successful releases, at least.

Alternatively, you could listen to the api:build entity's create event to know when a build starts, and the update event to know when its status changes - which I haven't used but presume would include a field for whether it succeeded or failed.

To subscribe to a webhook, you use the Heroku CLI:

heroku webhooks:add --include api:build --level sync --url https://example.com/hooks

This is outlined in more detail in the documentation here. The payload you receive will look like this.

This isn't going to get it straight into Slack for you though, so you'll need a piece of middleware to do that. Here's a tiny Sinatra app I forked from a fork. It converts Docker Hub notifications into the format Slack expects for an incoming webhook. It should be pretty easy to edit it to make it convert Heroku webhooks instead. Then, voila, the tiny app itself deploys on Heroku, and that's the URL you stick in the webhook subscription above.

This is probably a lot more steps than you wanted. Someone may have already done it somewhere... but I can say that at least doing it the above way will give you a lot more flexibility to create your own customised workflows where you need them!


Is there a way to find out every resource that calls a lambda? I have a lambda that gets called from several place through out an large app. Is there a way to discover every resource that is calling/using/attached to that lambda?

if you want to check from where lambda is calling/using/attached you can check it form designer in lambda.

Dashboard of lambda having designer that show lambda is connected with API Gateway , alexa skill etc.

So you can check everything from designer of lambda although you can also use cloud formation stack if you have made it for lambda.

  Here Lambda is connected with cloud watch and amazon EC2

There should be api gateway and other component also


Managing branched DLLs in nuget Our company has a product line and most development happens on the trunk.  We had 3 releases slotted back to back with the last one being an update to the first one.  the first one is in 3rd party testing with the second in in house testing.  The development of the third is completed but due to the amount of breaking changes was done on a separate branch of the first two.  This code needs to be placed on nugget to prevent any checked in code from referencing the local developers computer.  I don't want to make a nugget package for this branch as it will just die/get pruned when it goes into internal testing.  I also don't want to override the current package as internal testing may require changes to be made, causing confusion in version numbers.

What is the best way to manage/publish branched DLLs using NuGet?  should NuGet even be used?

So what is happening is that the changed code just sat until internal testing was done (never a good thing as this is how code get stale quickly).  Once internal testing was ready, the code was then merged into the trunk causing it to be published into nugget under the trunk's feed.  Some business politics ensued and we finally got to a real single code base mindset from the product owner's perspective (which was the issue that got us into this situation).  Moving forward I don't anticipate this to arise.

Code can be branched out.  However, only code in the trunk will go to our internal nugget feed or our build/deployment processes.


CI/CD for microservices in monorepo on Gitlab I'm working on a project that consists of several different services in different languages (a React webapp, a Scala backend, some Python stuff etc). I currently have each of these services in separate repositories on Github, and build them separately using CircleCI.

Some people seem to promote mono repos. As I'm planning to migrate the code over to Gitlab I wanted to take the opportunity to understand how this could be made into a monorepo. I have three questions:

Assume I put all the services in the same Project (Gitlab repos are called Projects right?), can I then get the CI/CD system to build each component independently?
Can I have Gitlab emit several images into my container registry?
Assume I commit a change to one service, and want to make sure it works with my other services before I deploy, how do I best setup integration tests that condition the success of my commit on the interaction with my other services?

Monorepos are nice because it eliminates the technical constraints between multiple projects. This does however open the door to other complications within your repository (naming conventions, cross-team dependencies, merge conflict increases, etc.). I do not have any experience with CircleCI, but I will provide some input based on other CI tools I have used.

I don't see any reason why you wouldn't be able to. In your CI/CD system, instead of referencing your individual source repositories, you will use your main monorepo as the source repository and then reference the specific project directory within it.
It looks like as of GitLab version 9.1 this is supported.
In my opinion, this wouldn't be something that would happen on commit. You could do this, but I think it would slow down your development speed significantly and use up a lot of resources if you have many commits. The two key points to consider are:

Before every production release, you should run a full suite test on all projects in your infrastructure. This is the only way to be certain that all project dependencies are non-breaking.
Tests/provisionings must run in parallel. By doing this, your full repository testing speed will be the length of the longest test/provisioning in your repository. (As a side effect, this puts an emphasis on speeding up your slowest test)


LambdaA -> delay mins -> lambdaB I need to launch a LambdaA function and after X minutes I need to launch a LambdaB function with parameters passed from LamdaA

I have testing the following:

APIGW > LambdaA > SQS with delay  > SQS can't trigger lambda functions
APIGW > LambdaA > SNS > trigger LambdaB . In that case I can't add a delay

I need to do it serverless and without constant pooling for a queue or new messages.

Is there any other possibility to trigger a second lambda function with a "big" delay?

Use Step Functions. They allow you to coordinate lambda functions and have an inbuilt wait state


RBAC to secure Azure Table Storage access I want to limit my user assigned MSI to be able to read and write tables only in a storage account. When assigning roles on IAM blade in the portal I don't see any table related roles. Why is that? I don't wanna use any SAS.

For now there is no support for accessing Azure Table Storage with Managed Identities.

Supported services are described here:

Services that support managed identities for Azure resources

According to documentation, Azure Table Storage allows you to authenticate only by using account key. So SAS authentication is not solution here either.

Possible solution could be like this:

Create Azure Key Vault and store account key as a secret Assign
permission to this key vault to your Managed Identity Use Managed
Identity to fetch the key from key vault in a runtime
Use the key to Azure Table Storage

As additional step you can create separate Storage Account to store only tables. With this approach you should be able to restrict access to tables service properly.


When to use Apache Airflow (MWAA) in AWS instead of AWS Batch? I am trying to get a compare and contrast of various job processing services in AWS like AWS Batch, MWAA (Managed Workflow for Apache Airflow).
Both are used for processing jobs. Just trying to understand when we should go for Batch and when we should go for MWAA.

AWS MWAA uses Apache Airflow to create workflows and DAGS (Directed Acyclic Graphs) with Python to orchestrate complex, dependent tasks. AWS Batch is, as the name states, a batch processing services that utilizes docker containers. Batch allows you to manage instance types and container sizes to fine tune costs of workloads. Batch unlike MWAA does not utilize any orchestration service, so apart from the queue that it utilizes, there is no way to control the workflow of tasks.
If you have a job that requires complex workflows, AWS MWAA will reduce the complexity of managing those workflows. If you require large scale containerized processing for image analysis, data conversion, etc., AWS Batch will allow you to easily control your workloads to optimize speed and cost.
It should also be noted that AWS Step Functions is also a very efficient way to orchestrate various types of processing and application tasks. For simple tasks, there is very little overhead and management compared to AWS Batch or MWAA.


Importing non-default workspaces in Terraform I'm having a bit of trouble with Terraform.  I have a Terraform project which is shared by a few developers and uses non-default workspaces (i.e. named workspaces, not the default one).  I would like all the developers to share access to the Terraform state of this shared workspace the same way as if it was the default workspace.  I have set up remote state with S3, so the state file is singleton, but for some reason I can't get the remote state file to be recognized on each developer's machine.
Basically the workflow I would like is, when developer A pushes a change to Terraform (using terraform apply) then any other developer who runs terraform state list should see that change reflected in the deployed infra, and that this workflow should work for any workspace, not only the default workspace.
How do I accomplish this?

As the docs state, workspaces are for distinct resources:

Named workspaces allow conveniently switching between multiple instances of a single configuration within its single backend. They are convenient in a number of situations, but cannot solve all problems.

A common use for multiple workspaces is to create a parallel, distinct copy of a set of infrastructure in order to test a set of changes before modifying the main production infrastructure. For example, a developer working on a complex set of infrastructure changes might create a new temporary workspace in order to freely experiment with changes without affecting the default workspace.

If you would like all developers to see the same state, they have to use the same
workspace. If you want this to be the case for arbitrary workspaces, then it seems that this is an abuse of the concept of the workspace. It is possible, but I would strongly advise against it, to import resources from non-default workspaces into the default one, in some kind of automated way. You could also use the default workspace state as the non-default's remote state, but again, I would strongly advise against this, arguing that it introduces unnecessary complexity and fragility to your process.
If you want all developers to see the same resources in the state, then they should simply all use the same workspace. If they are not seeing the correct state, it is because the workspace or the remote state location that they are using is different.


What is a Platform Engineer? I've been called many things over my career, most of them polite - more recently the work to build CI/CD pipelines has been termed "Platform Engineer" or "Platform Engineering".  The risk of such a title is that you get typecast into the role of an engineer who develops platforms for use by Developers and doesn't necessarily include any of the Ops bit of DevOps.

In my experience, however, the roles have involved a great deal of Operations Discipline work such as designing Major Incident Management Processes, Alerting, and Monitoring solutions.

In the role it "feels" like it has a lot in common with Site Reliability Engineering, in that both roles are engineering roles somehow set apart from software engineering they seem to have a similar Google Trends Pattern though:

Is there a canonical definition of a Platform Engineer or is it like SRE simply an implementation of a DevOps capability within an organisation?

I think you would be hard pressed to get a straight definite definition of a Platform Engineer.

Where I'm currently positioned everyone on the team starts out with their own special role but very quickly the end-game seems to be to feel comfortable enough to be called, or to call ones self, a platform engineer.

When I was first told that officially a platform engineer the brief explanation of what that meant, for me at least, was that I was going to need to do a whole lot more defending of my platform.

I wasn't quite so focused on helping our developers directly (although that's still the goal obviously) but as the platform owners our jobs was mostly to make sure that our developers where being good citizens by making sure we designed the platform so that we could show them what they did wrong if something happened and we could give them all the information they needed to fix their code.

The obvious implication of this is that our platform is perfect, which it obviously isn't. But at least we can support the developers in a way that gets them working on a solution on their end while we're fixing up our side of the agreement.


What is the Maven versioning strategy for Continuous Delivery or Deployment? At the moment, we're doing "traditional" deployment, where every couple of weeks, we roll a release. We rely on Semantic Versioning, which has worked well for us.

However, we now want to release faster, using Jenkins Pipelines etc, but I'm not sure what the best versioning strategy would be. Not having any versions (only Snapshots) is a step backwards, creating new versions all the time seems to be a lot of work.

So, if you are doing CD with Maven, what's your strategy?

we now want to release faster

And from your comment:

  Basically, since the artifacts depend on each other, we need to release them in the correct sequence. And since there's a lot of them, it takes some time.

I think you're having an X/Y problem. Yes, versioning plays a role, but it seems like you would benefit from solving the source of your problem first, which is that you have to release something in the correct sequence; presumably you also have to release everything at the same time, with a large downtime of the whole system.

Try to reach a state where you can release new versions of each artifact in arbitrary order. This basically means:

Employing some form of Dependency Injection (which is a big word, but can be something pretty simple, depending on how your software works).
You need to be able to have multiple versions of each artifact on the server at the same time, and to switch between them without much effort. That "switch" should be be on the side of the "user" of an artifact, not on the artifact itself, if you can do so at all.
Wherever you can, it also is great to be able to run multiple instances simultaneously (obviously this is restricted for data management stuff, databases and such).

I'm not telling you how to do that or that it is easy - I don't know your architecture or technical constraints. But this should give you a general roadmap.

When you have that capability in place (at least conceptionally), then you will be able to release smaller parts more often, and a little more independently. And - to answer your actual question - sticking with SemVer is then not the worst you can do.


AWS S3 ACLs across accounts I have two AWS accounts: A and B (viewable from https://console.aws.amazon.com/organizations). I have given an S3 bucket in account A a policy that allows account B to do everything in that bucket. When I have an EC2 instance on account B upload a file to that bucket, however, account A cannot access it because the owner is account B.

Is there a way of resolving this without having account B specify an ACL that includes account A for every upload?

No.  The account that uploads an object into a bucket owns the object.  All the bucket owner account can do, unless granted permission by the object owner account, is delete the object.


Production grade methodology for alerts Background

Our code is written with:

Unit tests
End to end tests
Code review
Staging process
Deployment process

On the contrary, our alerts are just written and then modified occasionally manually. No quality process at all.

This process is reasonable for simple threshold checks. However, our alerts are sometimes built on complicated queries. Sometimes composed of ~20 lines of a query.

If we accidentally break an alert, it could expose us to production instability since we won't know if some logic or component breaks.

The question

Is there a recommended methodology for validating the quality of complicated alerts?

P.S.

We're using Splunk alerts

DTAP

As monitoring is part of production it should be part of DTAP as well. This implies monitoring in Development, Testing, Acceptance and Production as well. If somebody modifies a check in production and it will silence some check and if the team does not get reported that there are issues then the customer will be impacted and that could be a huge problem. In short, if monitoring is part of production it should be applied to all stages of DTAP as well.

Testing

If custom scripts are used in splunk and these are written in Python or some other language, then you should also apply unit tests and integration tests. In most monitoring systems, a 0, 1, 2 and 3 exit code are checked so this should also be covered in the tests. If a monitoring is written in bash, one could use BATS and Powershell can be tested as well with Pester.

Why testing these scripts? Again, same reason as depicted in the DTAP section. Imagine that somebody breaks some monitoring script by introducing a typo and you do not get notified, then this could have a huge impact on the customer and also on the team. Imagine that you have to debug for a couple of days because some script is not working well, while that could have been prevented. So I advocate to apply unit tests, integration tests and even CI for this "simple" monitoring scripts.


How to shut down a stateless app gracefully? Quote from CF training on twelve-factor app’s processes are disposable:

  Apps should also shut down gracefully (but can also die unexpectedly).
  Cloud Foundry stops routing traffic nearly instantaneously to failed
  or shutdown instances.

Cloud Foundry apps are stateless. I don't understand how to shut down gracefully a stateless app. A stateless app can always handle kill -9?

kill -9 is used by the kernel to terminate stateful programs that are no longer responding (including to shutdown requests). To gracefully shut down a stateful app you would use kill -15 which instructs the application to call it's shutdown functions and do the requisite cleanup. If you app is truly stateless, there should be no difference in the results of kill -9 and kill -15, but technically, a graceful shutdown of this program would still be kill -15


Securely saving AZURE_STORAGE_KEY and storage account name in Linux systems The Azure CLI official documentation states that environmental variables can be used inside bash scripts to store AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY, like this:

Compared to AWS CLI this method is less secure since it stores the AZURE_STORAGE_KEY in plain text in the bash script.

AWS CLI, on the other hand, saves the key and the secret in encrypted format in user's home folder under ~/.aws folder.

Does Azure CLI provide similar functionality, or is there any better way to store the key safely in the system?

Instead of storing them as variables in the bash script themselves, can you store them as environment variables on the system from which the bash script is executing? This is a pretty common practice that will allow your scripts to execute on different systems without having to modify the script on each and every one.

Not sure what you mean by the AWS cli "saves the key and the secret in encrypted format", but if you look at the credentials file in ~/.aws, your access key and secret access key are both in plain text. I'm not sure how the Azure cli manages profiles, but AWS uses the AWS_PROFILE environment variable to distinguish which account the cli should execute as.


Using 32 bit for savings What are the downsides for CI servers to use 32 bit Docker images?

For example, using FROM i386/python:3.6.9-alpine3.9 instead of python:3.6.9-alpine3.9 in Dockerfile.

python:3.6.9-alpine3.9 (32 bit, 31.3 MB) is over 4% smaller than python:3.6.9-alpine3.9 (64 bit, 32.7 MB). The 32 bit images on Dockerhub are usually smaller than the 64 bit images.

When using 32 bit containers, I see no performance difference to the 64 bit containers. Using 32 bit images and containers save me space.

These are really the pitfalls as running a VM or traditional server with 32 bit CPUs.  Most notably is memory addressing.  32 bit CPUs can only access up to 4 gigs of memory.  This isn't something i would expect most containers have an issue with as they should be single, small, light applications.  One thing that you might run into is any binary built targeting specifically 64 bit architecture will not be able to run.  This is any application and their DLLs.  I don't think any web API will have issues with this, you will need to be aware of console applications.  Most dot net applications target any cpu allowing the runtime to target the correct underlying architecture.  There are applications that do target specific architectures and you will need to be aware of which ones they are.

In today's world, storage space is cheap.  The exception to this rule is embedded development (but I don't think that is the case here).  You need to ask yourself, is the extra 4% saved space worth the cost of me managing and ensuring that all the applications i'm running work on a 32 bit architecture.  Remember 32 bit applications will run on 64 bit architecture, but not the other way around.  Unless there is a specific need to only run 32 bit, like testing an emulated embedded environment (which is only 32 bit), I would just go with the 64 bit versions of everything.  To me, using 32 bit is solving a problem you don't actually have.


Windows Docker in swarm with Linux nodes connection issue I have the following setup for my docker swarm:

three Linux master nodes
one Linux slave nodes
Windows slave node

They are connected with an overlay network. Within the linux nodes everything works as expected.

But if I'm in a windows container(e.g. microsoft/nanoserver) and I try to ping a linux container with the internal dns name of the container. The DNS resolution works just fine, but the request times out. So far so bad. But it's getting stranger, because my linux container has two IP-addresses one is global loopback and another one which is in the same network.

Let's say the linux container has xx.xx.xx.2/32 as global lo and xx.xx.xx.3/24.
The internal dns resolves to the ip xx.xx.xx.2. This cannot be pinged from windows containers. But the address xx.xx.xx.3 can be pinged.
If I'm in a linux container, I can ping the xx.xx.xx.2 address. I also can curl the service(port 8080) of the windows container, within a linux container.

It's just weird. Has anyone experienced similar issues or can I manipulate the internal dns resolution of docker?

Also fun to note, it worked earlier. But after a swarm leave and join of the windows node it doesn't anymore.

Versions:

Windows Server 2016 - Docker version 17.06.2-ee-6, build e75fdb8
RedHat 7.4 (Maipo) - Docker version 17.12.0-ce, build c97c6d6

UPDATE:
 The windows node cannot reach a container which runs on the linux slave node all other containers can be pinged normally, but from the linux master nodes it is possible to ping the containers which run on the linux slave nodes.

I think Windows Server 2016 needs at least 17.09 Docker EE to support overlay networking and VIP's. I recommend updating it.

I was wrong, the latest version is indeed 17.06.2-ee-6 and should support overlay and VIP.

Second, ping isn't a good test, as it may not be available on the VIP or container IP depending on the setup (Docker has changed over time if the VIP does or does not respond to ping). I recommend using something that tests the Exposed container port, like curl. For testing you can just use something like bretfisher/netshoot which has common utilities like that installed.

Third, you could be resolving Swarm services on the network using the service name, not the internal hostname of containers. Depending on which network endpoint mode you use (VIP or DNSRR) in the service, this will resolve the correct VIP or the correct container IP for other containers on the same Docker network.

Also, you want a minimum of 3 manager nodes to follow raft consensus rules in a Swarm. 2 managers actually makes it less fault-tolerant then 1 manager.


AWS Site crashes every few hours Every few hours, my Laravel website on AWS Elastic Beanstalk crashes and throws the following error:

[Sun May 19 13:06:13.436195 2019] [php7:error] [pid 3431] [client]
   PHP Fatal error:  require(): Failed opening required
   '/var/app/current/vendor/composer/../symfony/polyfill-php70/bootstrap.php'
   (include_path='.:/usr/share/pear7:/usr/share/php') in
   /var/app/current/vendor/composer/autoload_real.php on line 66

[Sun May 19 13:06:31.485081 2019] [php7:warn] [pid 3459] [client] PHP
   Warning:
   require(/var/app/current/vendor/composer/../symfony/polyfill-php70/bootstrap.php):
   failed to open stream: No such file or directory in
   /var/app/current/vendor/composer/autoload_real.php on line 66

In-order to fix the error, I either go to /var/www/html or /var/app/current and use composer.phar install to fix the packages installed.

Unfortunately, this doesn't cut it. Once I fix the issue, it breaks again a few hours later with the same errors. I tried to fix the packages associated with the deployed zip file. But that's to no avail either.

Is there something I'm misunderstanding about the AWS architecture? I'm new to AWS, any help?

I resolved this issue by removing the vendor folder in the Laravel package.

Once the vendor folder is removed, the composer.json file is automatically used to run the installation. All necessary files are then downloaded and installed.

This removes the need to run the 'composer install' command post installation.


after automatic update jenkins fails with `InvalidBuildsDir: .../${ITEM_FULL_NAME}/builds does not resolve to an absolute path` I had a working Jenkins running as a Service on a Windows-2000 Server

I updated my Jenkins from 2.120 to 2.122  via Jenkins own update mechanism.

In the same run I updated some plugings too.

Now my Jenkins fails to (re-) start with this error:

jenkins.model.InvalidBuildsDir: /development/workspace/jenkins/${ITEM_FULL_NAME}/builds does not resolve to an absolute path
    at jenkins.model.Jenkins.checkRawBuildsDir(Jenkins.java:3060)
    at jenkins.model.Jenkins.loadConfig(Jenkins.java:3009)
Caused: java.io.IOException
    at jenkins.model.Jenkins.loadConfig(Jenkins.java:3012)
    at jenkins.model.Jenkins.access$1200(Jenkins.java:304)
    at jenkins.model.Jenkins$14.run(Jenkins.java:3104)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1068)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1102)
    at jenkins.model.Jenkins.<init>(Jenkins.java:904)
    at hudson.model.Hudson.<init>(Hudson.java:85)
    at hudson.model.Hudson.<init>(Hudson.java:81)
    at hudson.WebAppMain$3.run(WebAppMain.java:233)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:250)

I suspect that there is a problem to resolve the property ${ITEM_FULL_NAME}.

But on the other Hand I'm missing the drive letter (D:) in the path reported too.
The Jenkins instance is running from the same drive (D:).

What are my Options to track the Problem and fixing it?

I'm not very familiar with Jenkins on Windows, but I imagine your issue lies in the fact installing Jenkins on drives other than C:\ are more difficult to deal with. Since Jenkins is not running, you'll have to configure config.xml by hand.
Here's a post on stackoverflow.com that may help you. As they state, you'll probably need to find the Workspace Root Directory to D:\Jenkins\${ITEM_FULL_NAME}\workspace and Build Record Root Directory to D:\Jenkins\${ITEM_FULL_NAME}\builds (or whatever values are specific to your environment) in your config.xml.
The post also mentions the bubble help info that is useful:

This value may include the following variables:

${JENKINS_HOME} — Absolute path of the Jenkins home directory

${ITEM_ROOTDIR} — Absolute path of the directory where Jenkins stores the configuration and related metadata for a given job

${ITEM_FULL_NAME} — The full name of a given job, which may be slash-separated, e.g. foo/bar for the job bar in folder foo

The value should normally include ${ITEM_ROOTDIR} or ${ITEM_FULL_NAME}, otherwise different jobs will end up sharing the same workspace.


When do I run applications on a server in a pod and when to run them directly on the base operating system? I am trying to get started in the DevOps Field by setting up a small server currently with Virtualbox, later on for a small team. For my basic setup I plan to use gitea as self-hosted git and Jenkins as CICD tool which also provides a lot of extensions.
The server OS I chose is RHEL 8 since the systems on which I will setup up my DevOps system in a distant future will use this OS.
Now I find a lot of sources in the internet that point me to install both tools using containers combined with an orchestration tool like kubernetes. I have a certain degree of theoretical understanding of docker and kubernetes and also experimented a little bit with both tools.
My research also provided me information on the advantages of running Jenkins in a pod:

only required in certain situations
sometimes several instances of Jenkins for different projects can be required to run in parallel

At least were that the points that I found on different blogs websites that discussed this topic for Jenkins.
For Gitea I was not able to determine reasons why it should be run in container inside a pod in kubernetes but there is a helm-chart managed by the community of it. Also I found several sources on the internet telling me helm-charts are bad, don't use them.
To build, test and deploy applications (which will be Rest applications with react/node-js frontend and different backends) I will use container and kubernetes to orchestrate the container.
Could you help me understand why I should decide for or against containers/pods for the different applications and maybe even provide reliable sources, since most information I found are just opinions spread through the internet.

We're using a self-hosted gitlab, nexus and Jenkins (controller) and both are hard installs on CentOS7. Jenkins agent is a remote CentOS7. That approach made the most sense to us as opposed to breaking out the whole automated pipeline shebang to deploy gitlab/jenkins/nexus. Gitlab/Jenkins/Nexus are part of the foundation of our CICD infrastructure so it made no sense to make them ephemeral by using container, scaling or orchestration approaches. The actual software components we build & deploy are ephemeral (ie: a new build / config & deploy many times a day) and that infrastructure building, deploy & config is pretty much automated from scratch.


Terraform variable Validation is it possible to use validation of a variable type equals list(string)?
My example:
main.tf
  name            = "tf-prd-sg-foo"
  image_id        = "ami-00e788542ee66c64f"
  instance_type   = "m5.xlarge"
  security_groups = ["sg-000000", "sg-00000000"] # Example of SG only to contextualize
}

vars.tf
variable "security_groups"  {
    type        = list(string)
    default     = []
    description = "(Optional) A list of associated security group IDS."

    validation {
      condition = can(regex("^sg-", var.security_groups))
      error_message = "The Security Groups need starts to \"sg-\"."
    }
}

When I try execute returns the following error:
[terragrunt] 2021/01/05 17:12:10 Running command: terraform plan

Error: Invalid value for variable

  on vars.tf line 21:
  21: variable "security_groups"  {

The Security Groups need starts to "sg-".

This was checked by the validation rule at vars.tf:26,5-15.

[terragrunt] 2021/01/05 17:12:12 Hit multiple errors:
exit status 1

When validating an input variable of a collection type, we typically want to write a validation rule that tests if something is true for each value within the collection, rather than true of the collection as a whole.
You can write a "for each" sort of condition using Terraform's for expressions. If you are using Terraform v0.14 or later then you can write it concisely using the alltrue function along with a for expression that produces a true or false result for each of the items, like this:
  validation {
    condition = alltrue([
      for id in var.security_groups : can(regex("^sg-", id))
    ])
    error_message = "All security group ids must start with \"sg-\"."
  }

If you are using Terraform v0.13 then you won't have the alltrue function available to you yet, but you can get a similar result in a clunkier way by using the optional if clause of a for expression to filter out all of the valid ids and then checking whether the resulting list is empty -- that is, whether there are any remaining items that were not valid:
  validation {
    condition = length([
      for id in var.security_groups : id
      if !can(regex("^sg-", id))
    ]) == 0
    error_message = "All security group ids must start with \"sg-\"."
  }

This second example is a bit more cryptic to understand because it's using some language features in an atypical way, so I'll break it down into smaller parts to explain:

[for id in var.security_groups : id if !can(regex("^sg-", id))] filters the elements of var.security_groups to keep only the ones where the expression is true.
!can(regex("^sg-", id)) uses the ! operator to turn your original expression that decides if the value is valid into an expression which decides if the value is invalid. Consequently the result of the for expression will be empty unless there's at least one invalid value.
Finally, the length( ... ) == 0 around the for expression says that the value is only valid if filtering out all of the valid values produces an empty list, which conversely means that the list didn't contain any invalid values.


How to handle updating binary files on a dev server with git I have a dev site on a server which is synced with my git repo. The git repo does not track the media/ folder.

I want to update the content on the dev server to mirror the live server, so will copy over the database and the media/ folder from live to dev.

My worry is when I do this, git will pick up the changes in the filesystem with the new media/ files and won't allow me to do a git pull.

What's the best practice to handle this?

Git doesn't consider the untracked files for making decisions about accepting or rejecting a git pull operation, so you should have no worry about it.

If still in doubt you can easily verify it: just pull another copy of the same git repo in some other location - for test only - mess with untracked files in it and pull again.


Assign bug from different repo when creating a pull request Say I have 2 repos...

And I have a work item (or bug, or whatever) in one repo, and my code in another repo (yes, I know it's silly, that's how my group rolls for historical reasons).

Is there a way to assign a bug from one to the other? So that when I create a PR in Repo1, I can link to a bug in Repo2?

There are actually many ways to link work items to other objects in Azure DevOps.  Microsoft's documentation is pretty extensive on this topic.  However, for the particular task at hand, linking code commits (I'll assume Git is in use for this example), can be done in several different ways.

In your commit message - By including the work item ID in the commit message prefixed with the hash mark # (i.e. #12345), when the commit is pushed to the server, the work item and commit are linked automatically.  Then, when this commit is part of a pull request that is created, the linkage is made visible as part of the pull request.
Within the pull request during creation, or afterward when edited by the creator, you can add work items directly to the pull request by searching either by ID or by keywords.
Several other methods can be discovered through editing a work item or right in Visual Studio.


CodeReady Container for OpenShift won't start on Windows 10 Two different things might happen:

1 - On the output, get the following messages

ERRO unknown error occurred
ERRO unknown error occurred
ERRO unknown error occurred
ERRO unknown error occurred

2 - On ~.crc/crc.log, see the following errors:

Error dialing TCP: dial tcp [removed]:22: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or es
tablished connection failed because connected host has failed to respond.

Disable any VPNs on the computer
Create a virtual switch named crc

It took me some time to figure these out, so sharing


Is it possible to pass environment variables to multibranch pipeline job in a time of generating it via DSL? I want to pass some environment variables in the time of generating the multibranch pipelines, so upon the build start, it will already have these variables.

The only thing I found was this envInject plugin which is not supported by multibranch DSL generation.

https://jenkinsci.github.io/job-dsl-plugin/#plugin/envinject

Does anyone know the way to do that?

You can use Jenkins DSL and pass the environment variable as mentioned in below links:

https://jenkins.io/doc/book/pipeline/syntax/#environment
https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#withenv-set-environment-variables


How to only allow API Gateway requests to reach our EC2 instances We have multiple EC2 instances hosting our micro-services. The autoscaling group of servers has an ELB. All traffic is routed via the AWS API Gateway. The problem is that the ELB has its HTTPS port open to the world.

How to protect our servers, so that traffic is only allowed to pass through API Gateway?

Since November 2017, it is now possible to directly interact with servers in a VPC \o/

See:

https://stackoverflow.com/questions/32671394/can-i-specify-http-endpoint-in-a-vpc-as-resource-in-aws-api-gateway
http://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-private-integration.html


How to restore default Jenkins plugins? I do a lot of testing in my Jenkins VM which I have on my local laptop. I found this time-consuming to remove Jenkins and installing again from scratch. It would be better if I have a simple way to restore my Jenkins.

Is there a way to remove all the plugins that I have installed manually and keep the plugins that were installed during Jenkins installation?

One could also use docker Jenkins LTS. If one wants to start from scratch, just remove the mounted Jenkins home folder and there will be a clean Jenkins in no time.

https://github.com/jenkinsci/docker/blob/master/README.md


Sharing Gems between Chef and System Ruby I know about the chef_gem resource obv, but in the interests of good system hygiene I would like to avoid as much as possible duplication, so I would like to share Gems between the embedded Ruby within Chef, and the system-wide one. Additionally I have become aware of a troubling situation: the Gems I have in the system Ruby come from the known-good local Yum repo and are installed via RPM over the LAN whereas chef_gem goes off to the public Internet, and recent events in the Python community have made me very concerned about this. I have tried setting ENV['GEM_PATH'] in my recipe both before and inside a ruby_block but this appears to have no effect, I get a failure on require. What am I missing?

To take your points one by one:

  but in the interests of good system hygiene I would like to avoid as much as possible duplication

Can you ensure your system ruby will evolve at the same pace as the Chef's ruby ? Main point of chef coming as a bundle with its own ruby comes back from Chef 10 which was installed via a simple gem command and was usually a problem switching between ruby needed for applications and ruby needed by chef, the same goes for gem dependencies. I would advise against mixing system and chef ruby.

  the Gems I have in the system Ruby come from the known-good local Yum repo and are installed via RPM over the LAN whereas chef_gem goes off to the public Internet

While I understand and agree with the worrying about recent pipy events, there's two things here:

Your system ruby install gems from yum repository because your system ruby's gem command has been tweaked for that, that's specific to your distribution. -
Chef's ruby use usual ruby method going to rubygems.org, as any ruby installed by rbenv or any other way than yum install would do.

If you really want to use a local repo you'll have to set up a gem server (alternatively, some artifact repositories can handle that also) and set-up the sources for gem using chef's gem sources in /opt/chef/embedded/bin.

If you're using a chef client above 13.0 your can use the rubygems_url in client.rb to tweak that:

rubygems_url

  The location to source rubygems. It can be set to a string or array of
  strings for URIs to set as rubygems sources. This allows individuals
  to setup an internal mirror of rubygems for “airgapped” environments.
  Default value: https://www.rubygems.org.
  Changed in Chef Client 13.0.

And finally:

  I have tried setting ENV['GEM_PATH'] in my recipe both before and
  inside a ruby_block but this appears to have no effect, I get a
  failure on require

Chef omnibus package hardcode some values specially to avoid interfering with the system ruby, I think the GEM_PATH environment variable is ignored when searching for gem you require within the recipe code. I don't think there's a workaround for this, and again you're heading to a compatibility problem on mid term as you'll have a drift between chef's ruby and system ruby.


Necessity of mirroring git repos for small team considering switching to Azure Question:

  Are there any benefits to mirroring our git repos outside of Azure devops for a small (~7) team of developers who all work in the same office?

Background Information

I'm a developer who also wears the "DevOps" hat when he has time to do so (all the time right now). We're looking at potentially moving from our self hosted and maintained instance of Gitlab to MS Azure DevOps. My manager has stated he's interested in keeping a "forest of git nodes" for a couple different reasons.

One was access control (don't allow certain people to access node 'A')
One was that we don't want all of our code in Azure for some reason
One is so that we have extra backups of our code base
One is because "that's what git was made for, several distributed remotes. That's how Linux does it"
If Azure goes down we can still commit to the Gitlab repo and push that up later.

I think it's easy to tell what MY opinion of us doing this is, but I'd like others input on pros vs cons of going about structuring our git repo's in a way like stated above. From what I can tell he's thinking about the repo structure in one of the two ways shown below:

My reasoning for wanting to not bother with the mirrored repo's are the following:

The entire teams works in the same office, so there's no distribution of associates over multiple geographic regions. I'm also not aware that it would be changing anytime soon.
Adds complexity to the development environment.
Increases the possibility of code being out of date on one (or both) remotes. (For some background, in times before I was hired source code was getting lost.)
Access Control is built into both Azure DevOps AND Gitlab, so we're not getting anything new by doing that.
MS Backup and data retention are MUCH MORE comprehensive than what we have going on currently. Moving to them would INCREASE our probability of recovering data in case of emergency.
Relatedly, MS SLA is much higher than anything that I can guarantee (99.9%). Switching to Azure gets us that SLA for "free".
Since we're using Git as VCS, we're already working in a "distributed" manner. We don't need a mirror / second remote to push code to when we have the code locally on our machines anyway. We would simply push when connectivity was restored.
Admittedly selfish but, I don't want to have to worry about keeping all these tools (Nexus, Gitlab, OpenProject, all of those in containers on an unsupported Linux server that I don't have direct access to) up to date and patched. I'd rather be doing what I was hired to do, which is program.

With all of the above being said, I'm still open to having my mind changed. I'm new-ish to the development field and am very aware that I could be missing something crucial on why we would need a mirror or multiple remotes.

As you have realised Git is decentralised by design.

Using LAN repos

If Azure DevOps goes down, or your office internet connection, you can still use LAN repos. These could be simple file shares. You can also use emails to share commits; nothing I've tried but Git has native support for it). I have however pushed and pulled commits directly from and to my colleagues computers using file shares (because our online Git provider back then was extremely bad and slow (and the repos were many GB).

If you don't have automatic sync (which you can setup using pipelines/CI) between Azure DevOps and GitLab then the most up-to-date repos will reside on the developers computers which makes it easy to push the repo to a new location when needed (simply add the new remote and click push). However, if some repo hasn't been used in a long time it'll probably not exist on any developers computer and you'll therefor not be able to access it.

If you want to be disaster-ready I'd suggest hosting an Azure DevOps Server instance on your LAN that's synced with the cloud. Azure DevOps Server is capable of processing the very same yaml pipelines as the online version. You'd be protected against much more than just Azure DevOps Services outages.

The maintenance costs of having Azure DevOps Server as a backup for a small team shouldn't be much different than having GitLab as backup.

I'm not recommending these solutions, rather I'm just discussing them.

You might want to have a look at the Azure DevOps Proxy Server.


Kubernetes serve static content with ingress I have a simple Django application and on a regular EC2 instance nginx is a reverse proxy serving static files and proxying requests to gunicorn/wsgi backend app.
Should I have a separate deployment and service for nginx which will be able to access static data on a volume to serve it as well as proxy requests to django pods or
Is it possible to configure ingress to serve static data without extra nginx deployment?

You can't configure your Ingress to serve static data (from your host files for example) alone because the Ingress resource is just a way to configure a way of understanding for Kubernetes on how to access a specific internal resource from outside the cluster.

An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.
-- Ingress | Kubernetes - https://kubernetes.io/docs/concepts/services-networking/ingress/

You must configure a web server (NGINX, Apache, etc.) with a mounted volume bound to your files. You'll also need a Service resource to publish the web server access internally. Then the Ingress resource will do the work.


Scanning fs in a Docker container is (much) slower than on host? Is scanning/searching the filesystem slower in a Docker container, than on the host? Seems a lot slower from my testing and sometimes I get an EIO scan error.

Specifically I am scanning a read-only (:ro) shared volume.

In this case, the workaround is I can the host first before sharing the volume in the container, it seems much faster.

Any other workarounds? Is my observation imagined or real?

I am running Docker version 18.03.1-ce, build 9ee9f40

on MacOS Sierra

Docker on MacOS runs on a virtual machine (docker-machine), and it's known that docker-machine volume mounts may perform worse than what you'd expect. It becomes worse especially when it comes operations concerning lots of small files, like scanning the filesystem.

Official Docker documentation also mentions this, and suggests some workarounds by lowering the consistency guarantees.

See: https://docs.docker.com/docker-for-mac/osxfs-caching/


Spark on a Docker swarm - workers won't accept jobs IMO the dashboard does not match the log error, or?

Spark master dashboard says (similar for memory as well):

  Alive Workers: 2
  Cores in use: 2 Total, 0 Used

Log:

17/07/21 20:48:45 WARN TaskSchedulerImpl: Initial job has not accepted any
resources;
check your cluster UI to ensure that workers are registered
and have sufficient resources

Assuming there was e.g. not enough memory, how to determine that? Maybe a symptom, the workers get here and now "dissociated" but then reconnect.

TLDR; set workers' memory to 4G.

I reduced the test to the build-in Pi calculation:

./bin/run-example SparkPi 10

And got the same result; apparently 1G memory is insufficient even for simplest experiments.
Increasing both master and workers memory to 4G and the problematic behaviour was not reproducible. However before the systems start computation the message still could appear but just once, seems it is printed in a waiting loop.


Port forwarding for Windows Server like SSH port forwarding I have a Windows server 2019 that have many VMs instances (running using HyperV). Each VMs has its own private IP and has some apps running inside and expose via ports.

It looks like:

VM1: running app nodejs port 3000, private ip: 10.10.10.1
VM2: running app php port 3001, private ip: 10.10.10.2

Now I want to setup a secure connection so my colleagues can access the apps inside each vm without expose them to internet, how can I achieve it?

If I have a Ubuntu Server and SSH I will use dynamic port forwarding like this article

https://www.digitalocean.com/community/tutorials/how-to-route-web-traffic-securely-without-a-vpn-using-a-socks-tunnel

Does Windows have something alternative?

I have did a lot of tutorials but it seem it does not work for me.

You can use a ssh reverse proxy from any terminal (that has the ssh bin) to communicate with your applications.

For instance try this on the client side:

 ssh -NL 3000:localhost:3000 root@10.10.10.1

This command won't print anything (except prompting for ssh key's password) and won't stop by itself. To kill the reverse proxy you can do ctrl + C

Now you should be able to curl your application the following way on the client:

curl 127.0.0.1:3000


What are best practices for securing connections between CI and artifact managers in the cloud? Bitbucket, github, circleci are examples of Continuous Integration and Continuous Deployment (CI/CD) tools in the cloud.

In the past, a lot of companies ran all services, like CI/CD and artifact managers, like maven, artifactory on premise. The last years more and more services move to the cloud.

One of the advantages of on premise is that one could put everything behind a firewall and then everything that runs behind it could connect to each other.

If companies decide to run CI/CD in the cloud and to store artifacts in another cloud, what security measurements could one apply? One of the thinks I could think of is restricting the firewall, but some services that run on Azure expose a lot of different IPs and then you are basically opening the firewall for everybody in the world.

As I know for Git and Jenkins configuration we use SSH key authentication and secure token for security purpose.

In your case you can't open all port to all then you can set revers proxy for your all URL using apache or nginx and forward traffic to port 80, So when request come first it's hit the port 80 and internally it will hit to your specific application port.


docker mysql container not running suddenly docker mysql container was working perfectl.But after restarting of the server and using the start command to run the conatiner it is not working.My other containers are all up and running as normal.But mysql shows status exited.If I do docker start conatiner id it takes the command.But I am unable to enter into the container .Please find the attachment also

Without seeing the specific error it looks like the container's runtime process is either crashing or exiting.  When a docker container's runtime process exits the container stops.
 You can get more information about the specific error by using the docker logs command:

docker logs 92455a993a54


Spin up an EC2 instance from within the application I am developing the architecture of this application and the way we need to scale is not just stateless API requests, but it's a logical split of responsibilities. In other words, we may only be able to work on data 1-5 while we need another instance to work on 6-10. We will need an undefined number of instances and would like a manager to spin up and down EC2 instances as needed.

While scouring the web I am not able to come up with anything good, are there any good articles or is there a term for this? I have found IAC (Infrastrucure as Code) but from what I see that is simply deploying additional services, I don't see anything as IAC spun up from within the application. Is there a term for this?

Use AWS API to spin up the instances. There are SDKs (Software Development Kits) for various languages, for example if you're using Python use boto3 library. SDKs for other platforms are also available: Tools to Build on AWS, including the command line tools AWS-CLI for use in shell scripts.

With the SDK of choice you'll simply call something like ec2.start_instance(...) to spin up your new EC2.

However I would question whether you really need standalone EC2 instances for your processing? Perhaps ECS Containers, especially Fargate, or even serverless lambda functions triggered from a work queue may be a better choice?

Hope that helps :)


Aws autoscaling group in private subnet, private communication? I have a scenario -
I have configured my frontend application in one autoscaling group and my backend application is in another autoscaling group. I'm using the load balancer also. But I don't want API call from front end to the backend over the internet.
How Can I make front end application in autoscaling group to communicate privately with backend application which also in another autoscaling group?

I am assuming that the backend application has been set up in its own subnet that has been configured for private IP addresses only.  If not I would recommend that you do that as a starter for 10.

With that in place, you could set up an internal application load balancer to work on private only and have your backend ASG use this.
from the AWS docs
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-application-load-balancer.html


Is it reliable to use Docker's hash as a mean to determine image's version between different platform Situation

I have a docker image on a Linux box. I push it onto a docker registry (AWS ECR).

Then, I use the image pushed onto that docker registry in a K8s pod.

Question

How do I know the docker image used for the K8s pod is the same as the one on the Linux box? Note that the tag is irrelevant. I set it to different things all the time.

Yes, you can use the repository digest for this. Note that while this is a hash of the container, it's a hash specific to the repository, and is NOT the image id, which is a separate sha256 hash.

Once you push or pull a docker container to/from a registry, it acquires some repository metadata, including the repository digest for that container. You can see that digest by running docker inspect <container-tag-or-id> and looking for the RepoDigests section of the json (or just pipe that command into jq '.[].RepoDigests' if you have it installed.

Now you can compare that with the docker pullable of the kubernetes pod you want to check by running this command:

kubectl get pod <POD> -o jsonpath='{.status.containerStatuses[*].imageID}'

If the hashes of both of those values match, then you can be sure they are running the same container.


Terraform: Sanitize a map that may be a map of strings or map of maps or combo of both I want to take a map such as this that is passed in as a variable
cidrs = {
  A = [1.1.1.1]
  B = [2.2.2.2, 3.3.3.3]
  Group1 = {
    C = [4.4.4.4]
    D = [5.5.5.5]
  }
}

and then within the locals section i want to sanitize it to the following
cidrs_sanitized = {
  A = [1.1.1.1]
  B = [2.2.2.2, 3.3.3.3]
  Group1 = [4.4.4.4, 5.5.5.5]
  C = [4.4.4.4]
  D = [5.5.5.5]
}

I have tried various methods of looping
everything i've tried so far has led to me trying to get each set of values as seperate variables and then merge them together example:
single = { A = [1.1.1.1], B = [2.2.2.2, 3.3.3.3] }
groups = { Group1 = [4.4.4.4, 5.5.5.5] }
grouped = { C = [4.4.4.4], D = [5.5.5.5] }
cidrs_sanitized = merge(single, groups, grouped)

and here's some of the stuff i've tried but failed to succeed with
groups = flatten([ for range, value in var.cidrs: tomap({ range = values(value) }) if !can(element(value, 0)) ])
single = flatten([ for range, value in var.cidrs: tomap({ range = value }) if can(element(value, 0)) ])

and
groups = {
  for range, value in var.ranges :
    range => can(values(value)) ? values(value) : null
}
grouped = flatten([for range in var.ranges : [
  for cidr_key, cidr in range : {
    cidr_key => cidr
  } if length(cidr) > 0
] if can(values(range)) ])

The merge always fails as it tells me the output are tuples and i can't figure out how to get terraform to show me the values of the locals grouped, groups and single when calling it as a module
Any help or points in the right direction would be appreciated, thanks.

Following the documentation a map is a collection of key/values with a constraint:

"...accepts any element type as long as every element is the same type" source

So the input map you have declared, is already an invalid map for Terraform 0.12 (at least)
Below the example of the error produced:
variable "cidrs" {
  description = ""
  type        = map
  default = [
    ["1.1.1.1"],
    ["2.2.2.2", "3.3.3.3"],
    {
      C = ["4.4.4.4"]
      D = ["5.5.5.5"]
    }]
}

When validating an error is raised:
$> terraform validate

Error: Invalid default value for variable

  on vars.tf line 65, in variable "cidrs":
  65:   default = [
  66:     ["1.1.1.1"],
  67:     ["2.2.2.2", "3.3.3.3"],
  68:     {
  69:       C = ["4.4.4.4"]
  70:       D = ["5.5.5.5"]
  71:     }]

This default value is not compatible with the variable's type constraint: map
of any single type required.

Been that said, if we can generalize A and B as "values which are lists" and Group1 as "values which are maps" and the input is a fixed type ordered we can use a tuple or  an object, this can be solve in a more readable (and maintainable) way. If so, let us know so we can extend this answer.


Mitigate false positives due to daylight saving time in CloudWatch anomaly dectector For monitoring, I set up CloudWatch alerts based on CloudWatch anomaly detection. Overall, they work quite nicely, but they got confused when the clock gets shifted (summer timer to winter time).

We have now recurring false-positives even though the recent time shift in the US and in Europe were weeks ago. I assume CloudWatch will eventually understand it, but then again, the clock shifts have on a regular basis, which will create some noisy.

Unfortunately, I did not find much in the documentation, only that CloudWatch provides to set the Time Zone Format. Before I leaved it empty, but now I set it to UTC.

Questions:

What is exactly the effect of setting the time zone? Does it have an impact on the prediction of the anomaly detector, or is it only for visualization and for other configurations such as excluding specified time periods?
Our user base is all over the world, but mostly in the US and in Europe. As this scenario is not too uncommon, I wonder if asking for one one time zone is even practical. What should be used? UTC? Or pick the time zone with the most users?

I understand that it difficult to avoid false-positives completely, especially, in the first or second week after a clock shift happened. Still, I wonder how to correctly setup up the anomaly detection model, so its prediction will be most accurate.

My current understanding is that it is not possible to avoid false positives. At least not if traffic comes from different time zones around the world (as in the context of this question).

AWS operates on UTC times, but this is irrelevant for the anomaly detection. It does play a role if you set timers, but it is not taking into account for the predictions of AWS. In other words, switching to UTC will not make false-positives less likely.

When thinking about it, it is hard to avoid false-positives. If a  time-shift in Europe or the US happens, you could take that  into account and predict the new traffic. But that requires extra domain knowledge. The AWS predictor only sees a  generic time series.

There are other events such as Christmas, which will impact the expected traffic. Again, with extra knowledge you could improve predictions. But the problem is too difficult for the AWS predictor, as it does not have that domain knowledge.

In our concrete case, we still kept the alerts, as they do have value even with the known false-positives. Humans with knowledge about the underlying system, can reject the false-positive. It is not ideal, but it is a trade-off that we are willing to make.

Note that other types of graphs are not susceptible for these types of problems. For example, error rates instead of total number of errors. It is not always possible, but if you can use rates, the system will be more stable in my experience.
AWS allows to set alerts on math expressions such as errors / requests. The advantage is that normalized rates tend to be agnostic of external events.


Wordpress on AWS, how to update only a theme In order to setup a flow from developer code to a public URL, I've been testing some flows and the current one does not work that good:

I have an AWS Pipeline that picks up the source code from CodeCommit, builds the Sage Theme using gulp and deploys the entire code into an Elacticbeanstalk application.

The bad part here is that I'm pushing the entire code again and again, all I really need is the theme folder. Plus, I need to re-activate all plugins every single time :(

What is the best approach for such scenario?

I'm not terribly familiar with elastic beanstalk, so I'm not sure if my method is at all approachable with your workflow, but I have a similar (non-AWS) workflow, and I can share how I do it.

I've managed to only push theme changes by separating the deployment of wordpress, and its mysql DB (both as a docker containers, in my case), and the deployment of the themes (which is a git repo).

This way I can separate/automate the deployment of wordpress, mysql, and my wordpress theme in a fairly loosely coupled manner.

Updating consists of only performing a git pull to the themes directory -- No other wordpress/mysql interaction occurs, so my plugins stay in whatever state they were previously in prior to the theme update.


503 Service Temporarily Unavailable using Kubernetes I am having some issue with creating ingress for a nginx service that I deployed in a kubernetes cluster. I am able to open the web page using port forwarding, so I think the service should work.The issue might be with configuring the ingress.I checked for selector, different ports, but still could not find where goes wrong. Anyone could help ? Thank you in advance.
# Source: django/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: vehement-horse-django-config
  labels:
    app.kubernetes.io/name: django
    helm.sh/chart: django-0.0.1
    app.kubernetes.io/instance: vehement-horse
    app.kubernetes.io/managed-by: Tiller
    enter code here

data:
---
# Source: django/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vehement-horse-django
  labels:
    app.kubernetes.io/name: django
    helm.sh/chart: django-0.0.1
    app.kubernetes.io/instance: vehement-horse
    app.kubernetes.io/managed-by: Tiller
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
      name:
  selector:
    app.kubernetes.io/name: django
    app.kubernetes.io/instance: vehement-horse
---
# Source: django/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vehement-horse-django
  labels:
    app.kubernetes.io/name: django
    helm.sh/chart: django-0.0.1
    app.kubernetes.io/instance: vehement-horse
    app.kubernetes.io/managed-by: Tiller
spec:
  replicas:
  selector:
    matchLabels:
      app.kubernetes.io/name: django
      app.kubernetes.io/instance: vehement-horse
  template:
    metadata:
      labels:
        app.kubernetes.io/name: django
        app.kubernetes.io/instance: vehement-horse
    spec:
      containers:
        - name: django
          image: "website:72cf09525f2da01706cdb4c6131e8bf1802ea6a3"
          imagePullPolicy: Always
          envFrom:
          - configMapRef:
              name: vehement-horse-django-config
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            limits:
              cpu: 1
              memory: 1Gi
            requests:
              cpu: 1
              memory: 512Mi

      imagePullSecrets:
        - name: some-secret
---
# Source: django/templates/ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: vehement-horse-django
  labels:
    app.kubernetes.io/name: django
    helm.sh/chart: django-0.0.1
    app.kubernetes.io/instance: vehement-horse
    app.kubernetes.io/managed-by: Tiller
spec:
  rules:
    - host: something.com
      http:
        paths:
          - path: /
            backend:
              serviceName: vehement-horse-django
              servicePort: http

i'm guessing the problem is that the name of the service port is not configured, and in the ingress you specified the http name and it can not find it.
name your port http , or give the port that you want to use (80) to the servicePort in the ingress file


Trying to migrate CI/CD process from Jenkins to Gitlab Runner + Docker on Windows I need to improve the CI process of my projects, which is basically composed of the following things:

a Linux server, that contains a Jenkins master
a VM with Windows 10, with a Jenkins slave, responsible to build one of my projects (that uses .NET core, .NET framework, Node.JS and nmake)
a Jenkins pipeline job that compiles the project (assuming the needed tools are already installed) -> this is used to generate final released versions to the customer
a Gitlab Runner on some Windows machines (configured as 'shell' executors) -> this is seamlessly used with our gitlab, for continuous integration and pipelines, to be visible on Merge Requests, for instance

Problems: we manually entered the VM above and installed all the tools needed to build the applications. If the projects require new tools, we need to manually enter the machines and install them prior to running the build jobs.

What I want: I plan to improve this CI environment with Docker (however I'm pretty new to it), to be able to easily scale the job runners. I'm thinking about using less Jenkins and more Gitlab Runner instances (configured as 'docker' executor), to be able to delegate the CI/CD processes to each '.yml' file versioned on each project.

To accomplish the above, I thought about some approaches:

Put a Gitlab Runner configured as 'docker+machine'

This spawns as many 'docker' runners as needed --> this is easily scalable

Problem: "Docker Machine" only creates Linux VMs, and I need specifically for this project a Windows environment for the build

Use a Gitlab Runner configured as 'docker'

This should be scalable, if it spawns containers for building on demand.

Based on this scenario, I have a couple of questions:

do you have any suggestion about how can I create a docker image that is able to build my project? I need to use Windows OS, .NET core, .NET fw, nmake, Node.Js. Do you have some specific images recommended for this task?
should I create a 'sdk' docker image (an image with all tools needed for building and testing) separate from my project source-code, and publish this image to Docker Hub, and 'consume' it on my gitlab runner? I mean, what's the best practice for this case, supposing my customer does not have a private Docker Registry?
have anyone tried this scenario (gitlab runner + docker on Windows) before, and have some feedback if it works or not?
Is my plan of decreasing Jenkins usage and using more Gitlab runner (for this project at least) feasible? I thought about this because we currently have to manage a "Jenkins Master" and all "Jenkins Slaves" machines. With this new approach, the 'master' would be actually the entire Gitlab instance, and the 'slaves' would be several GL runners deployed as I need, on which I would delegate the job to each project's ".yml" file;
Do you have a better approach on how can I improve to a scalable Windows project CI/CD process, other than using gitlab runner + docker?

have anyone tried this scenario (gitlab runner + docker on Windows) before, and have some feedback if it works or not?

I installed and registered a GitLab Runner on my Windows 10 laptop following the instructions from GitLab Install GitLab Runner on Windows

At first, I had trouble pulling the Windows container image:

PS C:\GitLab-Runner> docker pull mcr.microsoft.com/windows/servercore
Using default tag: latest
Error response from daemon: manifest for mcr.microsoft.com/windows/servercore:latest not found: manifest unknown: manifest unknown
PS C:\GitLab-Runner> docker pull mcr.microsoft.com/windows/servercore:1809
1809: Pulling from windows/servercore
no matching manifest for linux/amd64 in the manifest list entries
PS C:\GitLab-Runner>

After selecting the "Switch to Windows containers" option in my Windows Docker (in the top-level menu that opens from the Docker icon in the toolbar), I was able to docker pull the Windows servercore image.

However, I was unable to run GitLab Runner on my fully patched and updated Windows laptop:

 ERROR: Preparation failed: unsupported Windows version: Windows 10 Pro Version 1909 ...

It's too new!  As I read GitLab's Windows version support policy, GitLab should support 1909 for 18 months; but obviously that is not the case yet. 1903 is the most recent supported version.

However, as of January 2020, GitLab has Windows Shared Runners (beta)

I tried that and it works. Example .gitlab-ci.yml:

my_job:
  script: systeminfo
  tags:
    - windows

  Is my plan of decreasing Jenkins usage and using more Gitlab runner (for this project at least) feasible?

Sounds good to  me.

  Do you have a better approach on how can I improve to a scalable Windows project CI/CD process, other than using gitlab runner + docker?

No.  I think you're on the right path.


How to take a backup of Ansible tower? Currently we have an awx running on a VM using containers:

CONTAINER ID     IMAGE                        COMMAND                  CREATED             STATUS              PORTS                                                 NAMES
dfb9fa57a8eb    ansible/awx_task:latest      "/tini -- /bin/sh -c…"   2 months ago        Up 6 days           8052/tcp                                              awx_task_1
60dcb5340ca5    ansible/awx_web:latest       "/tini -- /bin/sh -c…"   2 months ago        Up 6 days           0.0.0.0:80->8052/tcp                                  awx_web_1
e80aaddee738    postgres:9.6                 "docker-entrypoint.s…"   2 months ago        Up 6 days           5432/tcp                                              awx_postgres_1
54f20decccfe    ansible/awx_rabbitmq:3.7.4   "docker-entrypoint.s…"   2 months ago        Up 6 days           4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   awx_rabbitmq_1
0e7ed8c75796    memcached:alpine             "docker-entrypoint.s…"   2 months ago        Up 6 days           11211/tcp                                             awx_memcached_1

What is the recommended way for backing up our awx instance:

a) Take an export of the VM using ovf?

b) Just copying the volume where the pgdata is held enough?

c) Running

docker exec -t <postrges_container_id> pg_dumpall -c -U postgres > dump_`date +%d-%m-%Y"_"%H_%M_%S`.sql

All you need is a backup of the postgres database and the projects (to be safe but they should be rebuildable by pulling them again from your VCS in most cases).

Exporting an ovf of you full vm is really an overkill IMHO.
Copying the volumes (pgdata and projects) is by far the simplest solution (the one I actually use). You simply have to put the folders back in place, and docker-compose up -d the correct version and you are set. But to make sure your backup is consistent, you need to stop AWX before copying the dirs, especially for postgres.
Dumping the postgres database and copying the projects will let you do a backup while AWX is running. For your recovery, you will have to first pop-up a posgres container and reinject the dump before starting all the other containers.

Keep in mind that AWX is the dev version of Tower. As such, there is no real "recommendation" for backups. So the above explanation simply reflect my own experience with AWX. The commercial Tower version embarks the backup in it's deployment playbook. I have never played with it.

If you did not experienced with an AWX upgrade yet, you should be aware that they are sometimes working but can be a real pain (export from old version / import to new version / re-set all the user passwords and jobs credentials). So a backup to quickly rollback when testing the upgrade is vital in this case.

Hope this will help you make a wise choice ;)


Is there a way to clone an ec2 from inside the ec2 terminal? I want to be able to back up my linux accadamy lab servers but they don't provide a way to to this.

I'm wondering if it's posible to clone my EC2 lab server from inside the EC2 to get round this?

Yes, it can be done.

You can either create an IAM instance role which allows the instance to use ec2 actions such as "create-image" or you can create an IAM user and create for this user access and secret keys and a policy which allows this user to create images in ec2 and then configure the access and secret keys locally on the server using the "aws configure" command.

Whatever method you choose from the above options, the next step would be from within the instance itself and assuming you have aws-cli installed in the instance, first run:

ec2-metadata -i to get the instance id.

and then run:

aws ec2 create-image --instance-id INSTANCEID --name NameOfImage --no-reboot --region REGION

And it will create an image of this instance in the region you provided without rebooting the instance.

The create-image command will create an image of the instance, also called AMI which then you can use to create new instances from.


Create dynamic resources using 'for_each' or 'for' I want to create multiple resources using this:
resource "group" "list" {
 g_name      = group_name
 p_name      = project_name
}

The input local variable is nested:
> local.input
{
  "group" = [
    {
      "name" = "group1"
      "projects" = [
        "project1",
        "project3",
      ]
    },
    {
      "name" = "group2"
      "projects" = [
        "project18",
        "project31",
      ]
    },
  ]

I am able to transform it into a better form:
> flatten([for grp_key, group in local.input.group: [for project in group.projects: { group_name=group.name, project_name=project} ]])
[
  {
    "group_name" = "group1"
    "project_name" = "project1"
  },
  {
    "group_name" = "group1"
    "project_name" = "project3"
  },
  {
    "group_name" = "group2"
    "project_name" = "project18"
  },
  {
    "group_name" = "group2"
    "project_name" = "project31"
  },
]

I can not use for_each because it is not map neither set.
How could I create multiple resources using the for like:
resource "group" "list" {
  for item in ... :
    g_name=item.group_name
    p_name=item.project_name
}

you need to use count, and with it you should be able to access item by index: https://www.terraform.io/docs/configuration/resources.html#the-count-object


Database Backup using jenkins I would like to take MSSQL server db backup. Is there any plugin available for it.

I don't know of any current plugins for doing this.  That may because a lot of backup functionality is built into MSSQL.

The easiest way to automate a backup via Jenkins may be to use a command line script to use a sql command:

Sqlcmd -E -S touch -Q" BACKUP DATABASE [DBName] TO DISK = 'C:\artifact_directory\my_db_backup.BAK'

You could also also avoid automating this via Jenkins and just create a scheduled backup directly within SQL Server.


Using Chef for multi-node operations I have an application I would like to configure using Chef that spans multiple nodes. Let's say that the process of doing it consists of

Do a thing on node A capturing the output
Do another thing on node B with that output
Back to node A now for some operation
Now on node B again
...

One way to do this would be to write a recipe that stored the stage it was on on each node and just keep running it on both nodes repeatedly until eventually it had a chance to do all operations on all nodes. But this is clumsy and won't scale if there's node C, D, etc. I obv know about notify to sequence dependencies within a single node, but that won't work across multiple nodes. I can't be the only person who needs this, so is there a mechanism or a design pattern/best practice/TTP for this style of activity?

In short, Chef may not be the tool to use here.

Chef is a convergent model, so you have to write your recipes in an idempotent way such as after some runs it will be at the desired state according to the others nodes around.

Your approach about storing the state sounds the way to go, and you'll have to use an external orchestrator to schedule the chef-client runs alternatively on A and B.
If you have a chef-server, push jobs could be the way to go sticking in chef ecosystem, main advantage I see with it compared to others orchestrators is a pull model not needing an incoming administrative port nor ssh on the target nodes.


Integrate iOS Xcode Build Node with Jenkins X + Tekton We are currently investigating in replacing our old QuickBuild CI setup with a modern cloud-native solution that is integrated into our k8s cluster. Jenkins X with Tekton (without classic Jenkins) sounds very promising.
We do not only have k8s applications though. Especially our iOS builds need to be executed on a Mac build node, and thus cannot run in k8s.
(How) is it possible to integrate a Mac build node into Jenkins X + Tekton?

I don't think OS X builds can be done with Tekton yet. For builds that need OS X, I would currently consider to use GitHub Actions with support for macOS actions.


Pooled Build Server and Multibranch development to stabilize the trunk? I'm a Dev in a big (40+ Devs) old (16+Years) SW project. We still work with SVN as source control but we are getting after automation as much as we can.

One problem that hounts us is, that 40 devs check in stuff into the trunk without a fair chance of running all the tests. Our test suites consists of JUnit unit tests, Junit integration tests that run inside our app server and test business processes and some test tool blackbox tests that run after the deployment. Unittests run 10Min, Integration tests 40Mins and test tool tests run all night long.

You can imagine that for each check in, not ALL the tests and most of the time also not all of the Integration tests are run. Resulting in regular test failures on our CI environment (thats the good part: we have one!)
But with 40 devs there is to much going on at the same time, so nobody knows for sure why a test failed. "Was it me? I changed something at another part of the app. This could not be the case, right...?"

I try to promote a branch based dev process with short(!) lived branches that could be testet before merging them into the trunk. Switching to Git is part of it but I would not be religious about it. Goal is that the trunk is always green. But for this I need an infrastructure that can support up to 60 branches.

I'm dreaming up a pooled build server system, where each team could request that a certain branch (maybe in a given revision/checkIn-ID) should be build and tested (maybe with a given testsuite). All the build requests get into a queue and are processed by the build server fleet. (Currently we have the problem that not enough hardware is available to scale the build infrastructure. Pooling the available hardware seems to be the best approach in my mind.)

What do you think about this approach?
How would you do this? Is https://jenkins.io/doc/book/pipeline/multibranch/ the right path down the road?

Am I going down the wrong path here? I know that CI says that everybody checks in into trunk at least every day. But, damn, there are so many side effects when 40 devs check in together and the software (especially the data model) does not prevent side effects between parts of the application. We have to start somewhere. But where?

Attempting to run your validation on branches, before being integrated into the trunk won't get you very far. The reason is that fundamentally such validations are performed in isolation, most often they will not actually reflect the trunk picture after the branch is merged.

The only case in which the branch validation is, well, valid, is if no other change (of any kind) makes it into the trunk between when the validation started (assuming the latest trunk changes are merged into the branch at that time) and when the branch is merged back into the trunk.

If any other change makes it into the trunk while the validation is in progress it won't be taken into consideration, leaving room for breakage when the branch is merged into the trunk. To prevent that you'd have to merge again the trunk into the branch - to pick up the latest trunk changes - and re-run the validation.

Effectively this means the branch validations (or at least the successful ones allowing the branch to be merged into the trunk) need to be entirely serialized if you want to keep the trunk stable. Which may be manageable for a while if you keep a reasonably small number of such branches (maybe multiple devs working on the same branch?). How many branches? Depends on how stable you want the trunk to be: if you want to always pass your integration tests, your serialized validations would need to run the integration tests. At 40 min each you can support max ~60 branches (assuming the merges are automated and can be performed 24x7, which isn't trivial). Without automated merges the number of branches will be a lot smaller. This won't scale, the branch lifespans will gradually increase, diverging more and more from CI and will eventually wear down your team.

The same reasoning explains why devs shouldn't actually have to run all tests if they are too lengthy - by the time they complete them the results may be invalidated by other changes committed to the trunk. Also - results tests done by developers are often not that relevant ("But it built fine in my workspace!").

There are only 2 possible approaches (that I know of) to get relevant validations:

the ones performed by the traditional CI tools - performed after the changes are committed to the trunk, reflecting the actual trunk reality, with all changes integrated. The only problem is if these validations fail - effectively blocking further development on the trunk until the culprits are found and the regressions fixed. Which may be (very) difficult in larger scale projects - as you already noticed.
the ones performed by gating CI tools - only 2 out there that I know of: Zuul and ApartCI (which is my baby). These validations are performed prior to merging changes into the trunk, but in an orchestrated manner, ensuring they remain relevant - taking into consideration all other changes in flight that will eventually be present in the trunk by the time a particular change is merged into the trunk. The actual change merging into the trunk is entirely automated, executed by the CI tool itself, immediately after the relevant validations are completed successfully. These are, IMHO, the only ones capable of guaranteeing trunk stability in larger scale projects.

Pooling resources is a good thing - give them to the CI tool - that's how you maximize their usage.


How can I get the first part of the API Deployment URL from inside cloud formation template? https://u1i7vf9mnl.execute-api.us-west-1.amazonaws.com/v1

How can I get the u1i7vf9mnl portion of the URL from inside of the cloudformation template as a Ref or Fn::GetAtt or some other way?

Did you try the variable ServerlessRestApi? It works for us like this:

Outputs:
    EncryptURL:
      Description: "Encrypt endpoint URL for Stage environment"
      Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Stage/encrypt"