Is it possible to combine jinja templates in ansible like this is possible with erbs in puppet? It is possible to include a subtemplate in another Puppet template.

  <%= scope.function_template("subtemplate.erb") %>

Yes, that is possible according to this post.

When a second jinja template was created and this was included in the base jinja it was called.

{% include "checks/subdue.j2" %}


Developing in Docker for Windows Basics Starting with Docker for Windows and C# applications.

I've been able to do some basic building and running of dockerfiles. However, I've run into the first application that doesn't run right out of the gate, and am uncertain how I would begin to troubleshoot it.

What are some of the techniques used when moving an app into a container (Windows image) for the first time?

The first thing you'll want to do is look for errors.  It sounds like you have a Dockerfile that is creating an image, but when you run the image it appears to not do anything.  There are two immediate things you can do to troubleshoot this:

Do not use the -d flag when running the image.  This flag runs the container in detached mode and will not display container output to the console.
Examine the logs from the container.  Even though the container does not appear to be running, it is most likely crashing on start.  You can verify this by using docker ps -a which will show you all containers (including ones that have stopped due to crashes).

Once you know what the error is you can look at whether this is a problem with your startup script or the Dockerfile.  Based on what you've described so far this sounds like you could be missing files in the right location, or the Entrypoint or CMD is incorrect.


How do configuration management tools deal with someone messing up configuration files manually? I'm researching a bunch of configuration management tools, like, Salt, Chef, Puppet,  Ansible, etc. and am trying to understand how they deal with someone manually changing a config file on the server directly:

A config file that is being directly managed by the CM tool, eg. the nginx/apache/Postgres config file
A config file that the system implicitly depends on, but is not being directly managed by the CM tool (i.e. it uses whatever defaults the underlying OS provides), eg. the rsyslog config file.

Do these tools snapshot / checksum all files in /etc (and similar) folders and alert the sysadmin whenever something has been changed behind their backs?

PS: My mental model right now is that of immutable config files & infra, like nix

If the file is marked as being managed by a configuration management tool, it deals with someone messing up a configuration file manually being messed up by a hapless admin by clobbering the configuration file using something like Jinja or ERB templates

While this can often feel frustrating to an old-school UNIX admin when their change is clobbered in minutes or hours by a configuration management tool, (and I have seen some use things like extended file attributes to hack around this) it is far superior in that it 1) enforces a cattle-not-pets mentality and 2) self-documents a deployment - a factor which is vastly undervalued considering that 5-10 years later when an upgrade of a system is needed, hundreds of config files may need to be scoured such as a host file, limits files, logrotate files, syslog files, systemctl defaults files, firewalld/iptables config files, selinux files, cron files, and much, much more may need to be considered using the "pets" deployment philosophy.

9 times out of 10 this comes from a lazyness/unwillingness to learn a newfangled configuration management technology. That being said, I highly recommend that if you are going to manage a config file using a configuration management tool, you use Salt, Chef, Puppet, Ansible, etc to flag the fact that you have taken over management of said config file so that the admin is not completely blindsided whenever possible and as often as possible.

You can also support additive configuration - EG, I have used salt to manage rsyslog.conf and some files in /etc/rsyslog.d/ but not all files. This allows for a baseline set of defaults, while still supporting some customization without (barf) using a configuration management tool - thus allowing you to "have your cake and eat it too".


how to checkout only specific folder from git repo((monorepo)) and build Hey folks I need some help on the jenkinsfile. Below is my usecase

This is the strcuture of my GIT repo:

root
  |->app1
  |   |->jenkinsfile
  |   |->dockerfile1
  |->app2
      |->jenkinsfile
      |->dockerfile2

I am having a monorepo, app1 and app2 in the root folder and when there is a change in app1 folder, only app1 should build and same for app2. I have defined the jenkinsfile in jenkins but when its build. its looking for dockerfile1 in root folder not inside app1.

P.S: I can see some thread where people asking the same question but the answer is in detail.

jenkinsfile:

pipeline {
    agent any

    environment {
        PIPENV_VENV_IN_PROJECT = true
        DEVPI_USER = '\'jenkins_user\''
        DEVPI_PASSWORD = '\'V$5_Z%Bf-:mJ\''
        WORKSPACE="${WORKSPACE}/app1"

    }
    stages {
        stage('Notify Bitbucket') {
            steps {
                bitbucketStatusNotify(buildState: 'INPROGRESS')
            }
        }

        stage('Build Environment') {
            steps {
                sh 'docker build -t app-builder .'
            }
        }

        stage('Test') {

            steps{
                sh 'docker run --rm app-builder pytest'
            }
        }

You can check out specific files or folders from a repo using git archive.  I've done this from a Jenkinsfile using a shell command like so:

sh("git archive --remote=${repo_url} ${ref} ${path} | tar x")

You will, of course, need to set the variables repo_url (the url of the remote repository), ref (the ref/branch/commit/etc. to check out), and path (the path of the file or folder within the repo to check out).


Where does CircleCI-2.0 store the ssh private keys? Aim

To deploy docker container using ansible using CircleCI-2.0.

Method

A private key has been created and added using the CircleCI-2.0 UI.

.circle/config.yml

version: 2
jobs:
  build:
    docker:
      - image: ubuntu:xenial
    working_directory: ~/project

Results

Running ansible results in:

Failed to connect to the host via ssh: Warning: Permanently added
Permission denied (publickey,password)

Discussion

The assumption was made that the keys would have been stored in ~/.ssh, but this seems to be empty.

  Is your target host accepting this key (when tried manually)?

Yes. Log in using ssh root@host -i key succeeds.

  Are you using the same user in the container?

Yes. The root user is used both manually as from the CircleCI-2.0.

Conclusion

In summary, the question is where does CircleCI-2.0 store the private keys that have been submitted using the UI? The ~/.ssh directory is empty.

I still don't get how your config.yml could launch ansible without a step part. Here is my findings, but I'm unsure it is what you're after, but that was too long for a comment.

According to the documentation here you have to add a add_ssh_keys with the fingerprint of your key as seen in the UI.

- add_ssh_keys:
    fingerprints:
      - "48:a0:87:54:ca:75:32:12:c6:9e:a2:77:a4:7a:08:a4"


jenkins credentials not used as expected when cloning git repo I am trying to setup a jenkins pipeline for my IaC code validation, for this I'd like jenkins to clone a gitlab (hosted) repository, for this purpose I setup a credential using username / key type, i used a jenkins user + a generated rsa key (same key on master and slave for simplicity).

When logged in the master or slave I can git clone using the following command and result :

root@jenkins-slave-1 [~]: su - jenkins
jenkins@jenkins-slave-1:~$ cd /tmp/
jenkins@jenkins-slave-1:/tmp$ git clone git@l-gitlab1.domain:salt/salt_states.git
Cloning into 'salt_states'...
remote: Enumerating objects: 1177, done.
remote: Counting objects: 100% (1177/1177), done.
remote: Compressing objects: 100% (483/483), done.
remote: Total 1177 (delta 676), reused 1153 (delta 665)
Receiving objects: 100% (1177/1177), 1.64 MiB | 0 bytes/s, done.
Resolving deltas: 100% (676/676), done.
jenkins@jenkins-slave-1:/tmp$

I can see the correct entry in auth log on gitlab instance, the one saying that a pubkey authentication was accepted.

BUT, when I run the pipeline (that is using the same credentials), it fails with this message :

Cloning the remote Git repository
Honoring refspec on initial clone
Cloning repository git@l-gitlab1.domain:salt/salt_states.git
 > git init /home/jenkins/workspace/salt_build_test # timeout=10
Fetching upstream changes from git@l-gitlab1.domain:salt/salt_states.git
 > git --version # timeout=10
using GIT_SSH to set credentials jenkins user's key (same key on master and slave(s))
 > git fetch --tags --progress -- git@l-gitlab1.domain:salt/salt_states.git +refs/remotes/origin/dev # timeout=20
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress -- git@l-gitlab1.domain:salt/salt_states.git +refs/remotes/origin/dev" returned status code 128:
stdout:
stderr: Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The gitlab auth log is showing that the authentication tried to use a password auth mode, not a pubkey. ...

Oct  8 11:49:23 l-gitlab1 sshd[18490]: Failed password for git from 172.27.200.46 port 56750 ssh2
Oct  8 11:49:23 l-gitlab1 sshd[18490]: Connection closed by 172.27.200.46 port 56750 [preauth]

Any ideas about what I am doing wrong ?

The error:

[Edit]
I added a screenshot as my explanation are not very clear :(

The Gitlab plugin requires an auth token, you're trying to use an SSH key.

https://github.com/jenkinsci/gitlab-plugin#global-plugin-configuration

"PLEASE NOTE: This auth configuration is only used for accessing the GitLab API for sending build status to GitLab. It is not used for cloning git repos. The credentials for cloning (usually SSH credentials) should be configured separately, in the git plugin."


What role do Configuration Managment tools play in immutable infrastructure? If I'm using immutable servers/containers, do I need tools like Chef, Puppet, Ansible, or Salt?  Those config management tools are designed to establish a configuration and then maintain it.

If I'm deploying immutable servers, should I use configuration management tools only for initial provisioning?

There are a few answers to that:

Something needs to build those immutable images. It is certainly easier to use old-school-style procedural scripting to build something when starting from a known starting state but this can still get very unwieldy over time (e.g. Dockerfiles), especially when you end up wanting a big matrix of different images for things like different versions of software, different environments, etc. Packer and other image build tools integrate nicely with Chef, Ansible, Puppet, Salt, and more.
Immutability is a spectrum, not a binary. Even in a "very immutable" deploy it is not uncommon to have some configuration files that need runtime management. This is also a place where you can use CAPS tools, though lighter-weight options like Consul Templates or etcd might make more sense depending on the overall infra. If you're doing immutable app servers but your database servers are more traditionally managed using Chef, then it might make sense to use Chef even on the immutable side for minor management tasks.
Zero-day management. Immutability is nice, but what are you going to do when the next OpenSSL 0day drops? If you have your build pipeline in a position to create immediate hotfix images and deploy them, then that's great. But many people will probably not have that kind of quick-turn ability.
Things that can't be immutable. The un-answer, but a whole infra is rarely 100% immutable. Things like database servers and developer workstations (yes, those are part of your infra) are somewhere between difficult and impossible to make immutable.


Are AWS Boto3 calls asynchronous? In the API documentation for Apache Cloud Stack for example, they put an (A) to designate asynchronous API calls.

In the Boto3 - Python client API for AWS - documentation as I can't see anything like this - so my question is whether its API is asynchronous or not.

Boto3 is a client for AWS API, so by definition it doesn't handle the synchronous or asynchronous behavior of the API call, that's the API endpoint which define that.

You can safely assume that any API endpoint marked as asynchronous will be asynchronous in any API client. Some SDK implement methods to wait for those asynchronous calls, some don't and you need to write your own handling (usually polling regularly the state) if you need to wait the end of a task started by an API call (creating a CloudFormation stack for exemple).


Access S3 files from Datacenter over VPN to VPC I am trying to access S3 files from non-EC2 machine. non-EC2 machine is connected to VPC over VPN, also can reach to other hosts in VPC.

Created VPC Endpoint for S3
Main routing table of VPC has route for S3 pointing to VPCe (pl-68a54001 (com.amazonaws.us-west-2.s3, 54.231.160.0/19, 52.218.128.0/17, 52.92.32.0/22))
S3 bucket has Static website hosting enabled
S3 bucket has policy for accessing files from VPCe (https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies-vpc-endpoint.html)

I am able to download (wget) file from S3 to EC2 in the VPC. When I try to do same (wget) from non-EC2 (which is in Data Centre), I get timeout.

Anyone know what I need to do to get this working ?

You need to use...

the Internet, from your data center, to access S3 directly (not via the endpoint) or
an AWS DirectConnect connection to access S3 directly (not via the endpoint) or
an HTTP proxy server inside this VPC to access S3 (via the endpoint)

...because S3 endpoints are not designed to be used across VPN connections.  They are only accessible from instances in the VPC where they exist.

  Endpoint connections cannot be extended out of a VPC. Resources on the other side of a VPN connection, VPC peering connection, AWS Direct Connect connection, or ClassicLink connection in your VPC cannot use the endpoint to communicate with resources in the endpoint service.

  https://docs.aws.amazon.com/vpc/latest/userguide/vpce-gateway.html#vpc-endpoints-limitations


Deploying software that depends on a virtual machine for non-tech savvy users About

I am developing a piece of software which consists of a Node.js application which controls a virtual machine that runs Android-x86. Also there are some pieces of the Android SDK like ADB the software depends on.

My current approach

Currently there is no smart way of deployment whatsoever at all. Everything has to be installed manually. Depending on the system I also get issues regarding networking and so on from time to time. I also created another question here where I first thought about using Docker.

My problem

While researching if I can solve this using Docker I came up with solutions that require Kubernetes as well. I want to deploy the software to non-tech savvy users as well. So I cannot afford to have Docker and Kubernetes as dependencies due to the complexity of the installation process.

My question

Is there any technology that I am not aware of that can simplify the setup process for the end user? The perfect scenario would be a virtual machine or container containing the whole environment that can somehow be converted to a single binary which can be installed/executed.

I argue that K8s+docker is a good approach even for a non tech person. For k8s you can use microk8s which is available on linux, windows and macos. It creates a local, single-node k8s cluster for you. You can write a script (if the target OS is linux or mac) and automate the installation of kubectl, microk8s, enable addons and install your component k8s yaml file. If the target OS is windows you have to see how they automate stuff and use that method.


Is there an organization that offers a Hashicorp Vault SaaS solution? There is a number of recommendations to harden a Hashicorp Vault deployment, but is there a SaaS solution, like CloudBees offers Jenkins to prevent that the hardening should be done manually?

It seems that Hashicorp itself offers an enterprise solution including 24x7 gold support called Hashicorp Vault premium enterprise.


AWS MSK: deploying all Kafka brokers in the same availability zone When using AWS MSK, is it possible to deploy Kafka brokers in the same AWS availability zone?

From what I read, AWS enforces deployments across availability zones. If you use three brokers, it will require three subnets in different availability zones. Is that true or is there a way to bypass that restriction?

Background: for most situations, deploying across zones is a good idea, but we are operating a subsystem where availability is not crucial and where all services are running in the same availability zone. Traffic in the same AZ is free, so we would prefer that if possible. If Kafka is unavailable if the whole data center behind that AZ goes down, would be acceptable. Most likely, the whole subsystem would not work in that scenario anyway.

I cannot back it up with links to the documentation, but my strong assumption is that it is not possible. AWS MSK forces cross availability-zone deployments. (In most cases, it is a good idea, anyway. I understand that the scenario in my question is not so common.)

The only option to stay in the same availability zone is to use only one broker. However, it is not recommended to run Kafka with a single broker. Therefore it is not an attractive option.


Is it better to perform delayed jobs using queues or at app level? I'm trying to scale up my push notification infrastructure. We currently use laravel app running on 2 ec2 instances horizontally scaled behind a load balancer, and each instance has a beanstalk queue engine that processes jobs created by laravel (including push notification jobs).

the problem is that this solution is not performant. Sometimes we send a marketing push to all our clients.. numbering in the thousands, and the time range difference between the first push notification and the last can be many hours.

I was considering what infrastructure would be good options. I considered AWS SQS, but it turns out that it's max delay job capacity is 15 minutes. In my business, customers can make pre-orders 20 hours before. So what we currently do is scheduled a delayed job 20 hours from now.

But then some people suggested that the delayed part should only happen at the application level (ie delay(work, timedelta(hours=20))), and that would allow me to use SQS anyways. Their reasoning is that delaying jobs is a business logic thing, and it's best handled at an application level and not an infrastructure level.

I'm not sure I agree.. and that's why I'm picking AWS elastic cache with redis. What do you guys think?

This sounds on the fence with opinion based but I think there's an important thing that narrow enough the problem to be answered.

It is still an opinion but, as far as you're horizontally scaling, instances may appear and disappear at will, keeping the delay in memory is thus not an option, so at this point you'll have to workaround this and bake something like a notifications queue/table to keep what is to be sent at which time and periodically check what is to be sent.

Once that is said, you'll need a central place somewhere to keep the needed notifications out of the app instance themselves and as such you'll need a place to store them, it could be a redis service, a database table or whatever, you'll be coding it anyway.

When working on AWS, the services provided are already redundant and on high availability without much effort needed on your part, as such I'd go with an application code storing the notification somewhere with the time to send it and a lambda checking periodically which notification has to be sent, the notification can then be sent through SQS to take advantage of the retries in case of delivery failure or you'll have to handle it in the notification firing system.

The store part being elastic cache or something else is a matter of taste and team knowledge in my opinion, take whatever you're more comfortable with.
For the part firing the notifications I'd use something separated from the app itself, another app or a lambda checking periodically the 'cache' (DB or redis or whatever) for notifications to be sent.

This allows the app to be sure the notification once declared and acknowledged by the cache is properly registered and will be fired when needed and retried in case of problem. A last case is the impossibility to deliver the notification and you may have to keep track of this in the application to avoid sending too much notifications ending as dead letter but that's sliding out of scope for this question.


Is web delivery for Node.js apps a thing? A browser fetches some HTML via HTTP/HTTPS or a file, which may result in fetching a bunch of other resources, including some JavaScript, and then it runs it in a browser environment.  I would describe this as a mechanism for delivering code to the browser.

Does a similar web-based delivery mechanism exist for Node.js apps?  Is there a way to get everything that a node.js app needs to run via HTTP(S) and run in some kind of "node" environment?  Sort of like a headless browser, but no HTML?

I'm asking this in the context of updating node.js apps.  I really like how a server is in charge of the code that the browser is running.  I think the same thing would be great for a certain class of node.js app.  It would be great if a node.js could auto-update itself in a straightforward and secure way.  This was the first thing that came to mind.

After thinking about this in detail, it seems necessary to containerize the application and use a mechanism like Docker to deliver the application in its entirety to the machine that will run the app.  There just needs to be some supporting mechanism for ensuring that the app is always up to date.  I know there are docker-compose and kubernetes, and things like Watchtower, but is this really the best way for just keeping the application up to date?  During development it seems like the mechanism is copying code (like through git) and installing dependent packages (like via NPM).  But this is a very developer-centric workflow.  Internally, I guess it makes sense to use docker. But for web-delivery, I think a workflow for deploying a node.js-centric application and keeping it up to date would use some other mechanism.  What is that mechanism?  If docker is the way, then maybe I just need to refocus my question to find out how to auto-update docker containers.  But I'm fairly certain that exposing the docker API directly isn't really a secure option for web-delivery.  I also don't even know how node-based apps are typically delivered.  The few that I have actually get installed via the OS's installer mechanism (Like an .msi on Windows) - and that seems very native.  Surely there must be a more web-centric approach.

So, how should a Node application auto-update itself (or receive a push update) via the web?  (like a browser-based app does)

I'm not aware of a project that provides that out of the box. For local development, it is common to have hot reload. If something at the file level changes, it restarts the server. That is somewhat related, as you could trigger the updates on the file system remotely. Installing libraries could also be simulated by directly updating the  node_modules folder.

Beside that NodeJs itself provides ways to run arbitrary code without process restarts. One way is to use eval (as part of JavaScript). The other is to reload a file. Normally, require caches the file after it executed it, but you can purge this cache (require.cache), and require the file again. That is what the decache library does.

In theory, NodeJs provides all the tools to allow remote code being applied. Of course, from a security perspective, you open yourself to all kind of attacks. Make sure requests come from a trusted source.

  But for web-delivery, I think a workflow for deploying a node.js-centric application and keeping it up to date would use some other mechanism. What is that mechanism? If docker is the way, then maybe I just need to refocus my question to find out how to auto-update docker containers

I would also see it less as a matter of updating a NodeJs application (driven by the application itself), but rather as  of a good deployment pipeline (driven from outside). In the end, deployment is nothing else than pushing changes to production.

There are a variety of deployment approaches for NodeJs applications:

Classic deployment on a server
Docker based deployments (e.g., Kubernetes)
Serverless (e.g., AWS Lambda, Serverless Framework)
Platform as a service (e.g., Heroku)

If you are not sure, the easiest to get started with are serverless deployments (supported on all major providers: AWS Lambdas, Google Cloud Functions, Azure Functions), or platform as a service solutions (e.g., Heroku). Kubernetes (as a popular framework for deploying Docker containers) is powerful, but has a steep learning curve.


Github rest api post label gives error for array input We are hosting GitHub enterprise for our development. I am able to access the reset api and create jira, create PR etc.

When I try to add label to my PR, it gives error for array.

curl -X POST -u githuser:gittoken https://api.github.mycompany.com/repos/team/repo/issues/560/labels -H "Content-type: application/json" -k -d '{"labels": ["bug"]}' -H "Accept: application/json"
{
  "message": "Invalid request.\n\nFor 'links/2/schema', {\"labels\"=>[\"bug\"]} is not an array.",
  "documentation_url": "https://developer.github.com/enterprise/2.13/v3/issues/labels/#add-labels-to-an-issue"
}

I check, bug is valid label.

curl -u githuser:gittoken -X GET \
  https://api.github.mycompany.com/repos/team/repo/labels
[
    {
        "id": 163461,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/+1",
        "name": "+1",
        "color": "c2e0c6",
        "default": false
    },
    {
        "id": 382069,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/Blocked",
        "name": "Blocked",
        "color": "fbca04",
        "default": false
    },
    {
        "id": 163462,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/Changes%20Requested",
        "name": "Changes Requested",
        "color": "cc317c",
        "default": false
    },
    {
        "id": 404926,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/Release%20Review",
        "name": "Release Review",
        "color": "5319e7",
        "default": false
    },
    {
        "id": 228780,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/Review%20Pass",
        "name": "Review Pass",
        "color": "009800",
        "default": false
    },
    {
        "id": 228781,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/Review%20Requested",
        "name": "Review Requested",
        "color": "eb6420",
        "default": false
    },
    {
        "id": 426113,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/Staging%20Bug",
        "name": "Staging Bug",
        "color": "d6021a",
        "default": false
    },
    {
        "id": 163457,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/bug",
        "name": "bug",
        "color": "fc2929",
        "default": true
    },
    {
        "id": 163458,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/duplicate",
        "name": "duplicate",
        "color": "cccccc",
        "default": true
    },
    {
        "id": 163459,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/enhancement",
        "name": "enhancement",
        "color": "84b6eb",
        "default": true
    },
    {
        "id": 163460,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/help%20wanted",
        "name": "help wanted",
        "color": "159818",
        "default": true
    },
    {
        "id": 163463,
        "url": "https://github.mycompany.com/api/v3/repos/team/repo/labels/wontfix",
        "name": "wontfix",
        "color": "ffffff",
        "default": true
    }
]

I tried with other combinations, like string only, string with coma etc.. but its same error.

Actual command:

From my understanding of the docs, you should directly send an array (which is valid JSON), without wrapping it in an object. That is ["bug"] instead of {"labels": ["bug"]}.

So, the proper request should be:

curl -X POST -u githuser:gittoken \
    https://api.github.mycompany.com/repos/team/repo/issues/560/labels \
    -H "Content-type: application/json" -k \
    -d '["bug"]' \
    -H "Accept: application/json"


Mitigating Maven Central risks as seen from the DevSecOps perspective AFAIK there are two official primary repositories for Maven packages (Java language):

search.maven.org offered by Sonatype Inc.
mvnrepository.com offered by a private person @frodriguez

Now obviously it makes sense to have a complete archive for backward compatibility purposes. However already as mentioned back 2013 in the interview with Sonatype's CEO, there are lots of downloads for components with known vulnerabilities. So many seem just to not care.

So is backward compatibility of so much value here? Why not take some action to reduce risks? Of course there is the OWASP dependency check plugin but I think on long term it is possible to implement a deeper risk prevention strategy.

Question: What could be a feasible way to motivate Maven and repository makers to establish some kind of early time warning, like issue special headers on download, or and additionally for example reject download of vulnerable components by default? This would in my opinion save a lot of time and electrical power if not to expect every build on the planet to do the check if this can be done centrally effectively.

UPD. In year 2017, over 80% of projects still seem not to recognize the risks, despite court cases.

What could be a feasible way to motivate Maven and repository makers to establish some kind of early time warning, like issue special headers on download, or and additionally for example reject download of vulnerable components by default?

Your concern, i.e. the meat of your question is a very valid one: how to improve security when downloading "stuff" from the internet. Maven, yes, but also other places like Dockerhub, Debian, Ubuntu etc..

I have no idea what stance the Maven developers have on this; I assume they have thought about it and come to the conclusion that it is not a good idea, for whatever reason. Generally speaking, I see no particular reason to place this responsibility on the repository, though.

Having separate responsibilities between componentes (here, Maven Central as one component) is a very powerful concept in IT. You find it in Unix command line tools, in OOP class design, or server architecture. In this particular case, it seems perfectly valid to me to let Maven Central simply be a huge repository of "stuff". Its main function is to be there and supply the packages.

Placing a security filter, if you will, between mvn and Maven Central is then pretty easily achieved. For example, a company I work for runs their own hierarchy of Nexuses; while the developers can pretty much do whatever they want on their laptops, the automated build environments have no internet access and must go through curated Nexuses. Each project has their own Nexus (for seldom used packages), and they chain up to a common intra-company global Nexus.

Developers, then, have no way to insert packages into those Nexuses, but a dedicated team reviews new packages and decides whether they can be used.

Note that this extends not only to security, but also, for example, to licenses. Depending on the policy of the company, not every open source license may be acceptable. So this can also be screened for.

Another big reason would be non-security related bugs. Say you find that X of your projects use package Y, and Y is found out to contain a serious bug, maybe with the result of being able to trash all your data. Would Maven Central remove that version from their repository? Certainly not! The bug may only appear in a very small edge case, which, by chance could be very relevant to your company. So, by having your own repository structure inbetween, you can flag the affected versions as "invalid", and thus enforce bug fixes traveling downward to your projects.

Finally, a company might want to phase out old versions of packages as a matter of routine, simply to keep up with the times. Using such a "filter" approach helps with that as well.

Back to Maven Central: why do they not do it? The question is not just a technical one, but also organizational. Who decides what is a relevant breach of security? Are we only talking about actual bugs? Are we also excluding packages that are safe in the hands of a responsible and knowledgeable user, or do we also by default remove packages that lead to possible holes indirectly (i.e., in a Unix package management, one may designate the telnetd package as insecure although it is, per se, perfectly fine).

So there you have it. Such filtering is a complex issue, and complex stuff is best distributed to different entities instead of forcing it on one central body.


terraform - how should I clone my code repo? I have a very simple terraform configuration. I boot up an aws instance of Ubuntu trusty and assign a security group.

My question is who is responsible of getting the latest code from my code repo? is it a bash script in terraform user_data argument or should I make a modified AMI that has a bash script to pull the code repo.

Basically my repo has docker configuration in it and all i need is to clone and do docker-compose -f env.yml up

You can achieve the clone of the repo and the execution of docker-compose in several ways.

You are right when you think that cloud-init could do that. Check this link for more information


How to locally resolve host name starting with variable slug? An app, running on my laptop in development mode, comprises several Docker containers, one of which runs a Django web app.

This web app expects to receive HTTP requests using a hostname with a variable leading part, e.g. "test-account.survey.com", in which 'test-account' is a slug representing a particular account, and the remaining string 'survey.com' is fixed.

Now I know this can be resolved in a Live environment, involving external HTTP requests, by using wildcard DNS servers [ https://en.wikipedia.org/wiki/Wildcard_DNS_record ].

But I'd like to set up and configure on my laptop a local DNS server for the sole purpose of forwarding hostnames as above to a fixed IP address, e.g. 127.0.0.1 and "passing through" any other hostnames.

Any recommendations? Thanks in anticipation.

P.S. The DNS server must support Windows 10 and Mac OS.

People often use dnsmasq for this.  You can wildcard sub-domains like this fairly easily for local testing using it.

Here's an example - https://qiita.com/bmj0114/items/9c24d863bcab1a634503 but there are plenty online.


How to avoid an avalanche of images in Openshift when security patches keep coming in? Our applications are based on tomcat, strictly speaking on tomcat base s2i image provided by RedHat. This is updated quite frequently, and we need to comply to some regulations and apply the updates. This basically means, that with every single update, we need to replace the base image of each of a few hundred ones deployed. There is about twenty something applications, each in a few different code versions, so cannot be simplified with tagging. I was thinking of extracting tomcat/JDK somehow to another supply chain, but there it gets complicated as well.

Get newest tag

How I mitigated it was by creating several bots. The first one uses dip to get the latest version of a specific docker image. E.g.:

go run main.go -image ubuntu -latest "xenial-\d.*"

to get the latest tag of Ubuntu Xenial. If the docker image does not exist in your local docker registry, then it will be pushed.

Update tag in several Dockerfiles

The second bot, updates the tags in the Dockerfiles. If the build is successful then it is merged. Currently an approval is always required, but if it works for a couple of months and the team has enough confidence then an auto merge can be done.

Update orchestration

The third bot updates the docker-compose.yml files, k8s files or whatever orchestration is used. If an approval is done then it will be merged into master. Again if there is enough confidence then this could be automated as well, but I always prefer some manual intervention if production is applicable. Then the team is aware that there was an update and could revert if needed. On the other hand if the monitoring works well then the team will be notified right a way and if updates are done frequently, i.e. at least once every two weeks then the team is trained and knows how to deal with this.


Terraform - referencing existing security group Is there a way I can associate existing security group to RDS that is going to be created or the only way is to recreate the security group so it can be added to the state file?

You can retrieve security group from data source

variable "security_group_id" {}

data "aws_security_group" "selected" {
  id = "${var.security_group_id}"
}


How to deal with untested features in the develop branch before deployment? Our current software deployment lifecycle looks like this (using git-flow);

We test locally or on a development environment.
We submit a pull-request to merge the feature into the develop
branch
Our testers test the feature.
** If rejected ** we create a new bugfix, which we test locally and submit a new pull-request to merge with the develop branch.
At the end of the sprint we create a release from the develop branch.
Deploy the release to the acceptation environment.
** When accepted by all the stakeholders **  We finish the release, thus merging the release branch into the master branch.
The master branch is then deployed to the production environment.

This used to work fine, but now we’re forced to create a new release at the end of each sprint, which means that the develop branch might contain  rejected features awaiting a bugfix at the time of release.

To make it more complicated, we have two repositories, one for front-end and one for back-end and they have to be in sync. i.e. our front-end cannot contain features dependent on, an api endpoint that has been rejected by a tester and won't be finished before the release.

I'm at a loss and could really use some advice.

[Sorry for the long comment, but I thought this question needed some context.]

You could use the same tools used in Trunk Based Development (TBD) to carry work-in-progress in the trunk (the only non-release TBD branch with a longer lifespan):

feature-toggles/feature-flags
branch by abstraction:

  "Branch by Abstraction" is a technique [1] for making a
  large-scale change to a software system in gradual way that allows you
  to release the system regularly while the change is still in-progress.

These would allow merging a feature in develop/master at any time, without the feature being actually (completed/functional and) enabled.

If technically these are implemented using configurable means then at any time the software can be tested both with and without a feature enabled, simply by flipping the config flag in the respective test environment. Otherwise special builds may be needed to enable them (still a lot easier than doing a branch merge with the feature's full/partial code).

And you could then run all your feature testing in develop/master to get your features enabled, one at a time.

But if you go in this direction - why would you even struggle with git_flow and run into all these problems? Just go TBD and CI/CD all the way :)


is it possible to connect an AWS autoscaling group with an application load balancer? I'm working with AWS cloudformation I have an autoscaling group for a couple of machines running an API server, and I would like to set up an application load balancer as a single entry point, so that client application only sees one URL. I was able to create the load balancer and the autoscaling group but I don't know how to connect them.

Resources:
  NodeLaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      AssociatePublicIpAddress: false
      # IamInstanceProfile: !Ref NodeInstanceProfile
      ImageId: !Ref NodeImageId
      InstanceType: !Ref NodeInstanceType
      KeyName: !Ref KeyName
      # SecurityGroups is like this on purpuse, this work, IGNORE the error message if any
      SecurityGroups: !Ref NodeSecurityGroup
      BlockDeviceMappings:
        - DeviceName: /dev/xvda
          Ebs:
            VolumeSize: 200
            VolumeType: gp2
            DeleteOnTermination: true

  NodeGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    DependsOn: LoadBalancer
    Properties:
      LaunchConfigurationName: !Ref NodeLaunchConfig
      DesiredCapacity: "1"
      MinSize: "0"
      MaxSize: "10"
      # TargetGroupARNs:
      #   - !Ref LoadBalancer
      TerminationPolicies:
        - NewestInstance
      # VPCZoneIdentifier is like this on purpuse, this work, IGNORE the error message if any
      VPCZoneIdentifier: !Ref Subnets
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-node
          PropagateAtLaunch: true
        - Key: Owner
          Value: !Ref Owner
          PropagateAtLaunch: true
        - Key: Department
          Value: AI
          PropagateAtLaunch: true
        - Key: Env
          Value: !Ref Environment
          PropagateAtLaunch: true
        - Key: Job
          Value: KubernetesGroupNode
          PropagateAtLaunch: true
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MaxBatchSize: 1
        MinInstancesInService: 1
        PauseTime: PT5M

  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: decrypt-loadbalancer
      # SecurityGroups is like this on purpuse, this work, IGNORE the error message if any
      SecurityGroups: !Ref NodeSecurityGroup
      # Subnets is like this on purpuse, this work, IGNORE the error message if any
      Subnets: !Ref Subnets
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-node
        - Key: Owner
          Value: !Ref Owner
        - Key: Department
          Value: AI
        - Key: Env
          Value: !Ref Environment
        - Key: Job
          Value: KubernetesGroupNode
      Type: application

Your CloudFormation template currently has the following resources:

A Launch Configuration
An Auto Scaling Group
A Load Balancer

Follow these steps in sequence to attach the auto scaling group to the load balancer:

Add a listener resource to your CloudFormation template, as described here. Set its LoadBalancerArn property to the load balancer's ARN.

Add a target group resource to your CloudFormation template, as described here.

Add a "listener rule" resource to your CloudFormation template, as described here. Set its ListenerArn property to the listener's ARN. In its Actions property, add a forward action & provide the target group's ARN like this:

Actions:
- Type: forward
  TargetGroupArn: !Ref MyTargetGroup

Add the target group's ARN to the TargetGroupARNs property of the auto scaling group, as described here.

If you prefer to visualize a chain of CloudFormation resources linking the load balancer to the auto scaling group, it'll look like this:

Here's a complete example: LB to ASG Example.


Application to run PowerShell scripts I've created a number of scripts in PowerShell that require a number of parameters to be manually entered.

Unfortunately, until we have our Jenkins server setup to support these additional tasks I need to pass these scripts to other technicians.

Ideally I would like an application that could accept the parameters via predetermined fields and dropdown menus then run the appropriate script.

Is there anything already available that would fulfill that need or will I need to knock something up myself?

I think the time and effort it will take to set up another tool that performs this type of work will be about the same as it would be to setup Jenkins. If you are looking into a different tool though, I would checkout Rundeck. It is a great tool for creating scripts and allowing other privileged users to execute them. I'm not sure if it will meet your needs for predetermined fields/dropdowns, but there are plugins available that may help.


Ansible service module running state utility Ansible v2.5 introduced a new "running" state for the service module, the doc is not saying anything about it.

  started/stopped are idempotent actions that will not run commands unless necessary. restarted will always bounce the service. reloaded will always reload. At least one of state and enabled are required. Note that reloaded will start the service if it is not already started, even if your chosen init system wouldn't normally.

I presume it checks if the service is running and starts it if not, but this is exactly what the started state does.

What differs the running state from the started state ?

It's deprecated in favor of started.

TASK [Test postfix is running]
**************************************************************
[DEPRECATION WARNING]: state=running is deprecated. Please use
state=started. This feature will be removed in version 2.7.
Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.


How to create dependency between modules in terraform I have 2 modules A and B.
I want a particular resource in module B to be created only after  a particular resource in module A is completed completely.

I was trying to achieve just that with this as output in the moduleA

output "alb_arn" {
  value = "${aws_alb.model-server.arn}"
}

Then in the main.tf i.e root i added this,, in the moduleB

  dependency                          = "${module.common.alb_arn}"

Then in the moduleB variable file referenced it like this.

variable "dependency" { }

And in the moduleB main,tf used it as depends_on like this

resource "aws_alb_target_group" "modelServer" {
  name_prefix  = "rqm-"
  port         = "${local.modelServer_container_port}"
  protocol     = "HTTP"
  vpc_id       = "${lookup(var.vpc, var.env)}"
  health_check = "${list(local.health_check)}"
  lifecycle {
    create_before_destroy = true
  }
  depends_on = [ "${var.dependency}" ]
  tags {
    Name = "rqm-modelServer-${var.env}"
  }
}

Error:

Error: aws_alb_target_group.modelServer: depends on value cannot contain interpolations: ${var.dependency}

What you've described here is the recommended way to pass dependencies between modules, but it relies on features introduced in Terraform 0.12. You appear to be using Terraform 0.11, where unfortunately these features are not available.

For Terraform 0.12 it would be idiomatic to write it using the "first-class expressions" syntax, like this (with these examples spanning the three modules just like you did in your opening example):

output "alb_arn" {
  value = aws_alb.model-server.arn
}

module "common" {
  source = "./modules/common"
}

module "other" {
  source = "./modules/other"

  target_group_depends_on = module.common.alb_arn
}

variable "target_group_depends_on" {
  type    = any # only the dependencies matter, not the value
  default = null
}

resource "aws_alb_target_group" "modelServer" {
  # ...

  depends_on = [var.target_group_depends_on]
}

The main missing feature for the above in Terraform 0.11 is that it only supports resources in depends_on, and not other kinds of object like variables.

However, it's possible to work around that by introducing an additional do-nothing resource that exists only to be a dependency:

output "alb_arn" {
  value = "${aws_alb.model-server.arn}"
}

module "common" {
  source = "./modules/common"
}

module "other" {
  source = "./modules/other"

  target_group_depends_on = "${module.common.alb_arn}"
}

variable "target_group_depends_on" {
  type    = "string" # (because "any" isn't supported in 0.11)
  default = ""
}

resource "null_resource" "target_group_depends_on" {
  triggers = {
    # The reference to the variable here creates an implicit
    # dependency on the variable.
    dependency = "${var.target_group_depends_on}"
  }
}

resource "aws_alb_target_group" "modelServer" {
  # ...

  # Marking the null_resource as an explicit dependency
  # means this indirectly depends on everything the
  # null_resource depends on.
  depends_on = ["null_resource.target_group_depends_on"]
}

Note that in Terraform 0.11 the depends_on argument takes a list of literal strings containing resource addresses, not interpolated values. That was the specific reason for the error message you saw when you tried your example, but a reference like "var.dependency" would not have worked there either because only resource addresses are accepted as explicit dependencies in Terraform 0.11.


Use Terraform to manage ssh-keys in GCE I use Terrafrom and manage access to VMs in GCE by changing VM metadata.

Before I put it to string, but it not good, so I try to put them to variable and use it. But I faced problem. This is my code:

variable "ssh_keys" {
  type = "list"
  default = [
    {
      user = "atolkachev",
      key = "ssh-rsa my_key atolkachev"
    },
  ]
}

resource "google_compute_instance" "instance" {
  zone = "${var.region}-${var.zone}"
  project = "${var.project_name}"
  metadata = {
    ssh-keys = join("\n", [for user, key in var.ssh_keys : "${user}:${key}"])
  }

When I execute it, I get error:

Error: Invalid template interpolation value

  on main.tf line 24, in resource "google_compute_instance" "instance":
  24:     ssh-keys = join("\n", [for user, key in var.ssh_keys : "${user}:${key}"])

I tried to find solution, but failed.

There was used wrong type for variable, should be:

variable "ssh_keys" {
  type = "map"
  default = {
    "atolkachev" = "ssh-rsa my_key atolkachev"
  }
}


How can secrets be distributed in a docker swarm? Currently I use env and sql files so that docker compose can load secrets from there and build sql database docker instances with specific user/password. The question is, how can this be ported to docker swarm? Docker secrets seem to only provide secret files over encrypted channels, but not set environment variables or be able to configure docker builds.

Once the secret reaches a manager node, it gets saved to the internal
  Raft store, which uses NACL’s Salsa20Poly1305 with a 256-bit key to
  ensure no data is ever written to disk unencrypted. Writing to the
  internal store gives secrets the same high availability guarantees
  that the the rest of the swarm management data gets.

https://blog.docker.com/2017/02/docker-secrets-management/


How do I install Terraform libvrit_cloudinit? I have searched all day trying to find a document on how to install the libvirt_cloudinit provider for Terraform.

When I execute a terraform plan I encounter this error message:

Error: libvirt_cloudinit.commoninit: Provider doesn't support resource: libvirt_cloudinit

Try to use libvirt_cloudinit_disk instead of libvirt_cloudinit. I think it is deprecated.


Running AWS lambda in serial (for a given parameter) I'm not sure if this is the best place to ask this. But we've currently got something in Microsoft Azure where we have their serverless functions (Azure Functions) running in serial based off a particular key, say a user ID.
So I want these functions to be called in parallel for different users, but in serial for each individual user. I spent ages trying to do this with locking behaviour in the function logic but it was always causing issues until I discovered with Azure, I simply make my Azure Function trigger from Service Bus, and use the userId as the Service Bus session id. Then Azure makes sure only one 'session' is running at once.
We're looking at moving stuff over to AWS and Lambda functions, but there's so many services - AWS Firehose, Simple Queue, CloudWatch Events, Step Functions, Kinesis Data Stream. In my initial searching I couldn't find any references to something similar (but everything is called different things, and I can spend hours reading about a service and find it's totally nothing to do with what I'm trying to accomplish)
Does anyone know if any of them offer that kind of "serial for a given property" kind of behaviour out of the box?

Use SQS with Lambda to accomplish this.
SQS supports 2 types of queues: Standard & FIFO. FIFO is appropriate for your use case. Create a FIFO queue & attach a Lambda function trigger to it. This causes the Lambda function to be triggered when a message arrives in the SQS queue. When you send a message to a FIFO queue, set the userId as the message group ID. This will ensure that messages from an individual user are only processed serially but messages from different users are processed in parallel.
Here's the relevant section from AWS Lambda documentation:

For FIFO queues, Lambda sends messages to your function in the order that it receives them. When you send a message to a FIFO queue, you specify a message group ID. Amazon SQS ensures that messages in the same group are delivered to Lambda in order. Lambda sorts the messages into groups and sends only one batch at a time for a group. If the function returns an error, all retries are attempted on the affected messages before Lambda receives additional messages from the same group. Your function can scale in concurrency to the number of active message groups.


Can "When" like keyword be used in scripted pipeline? stage('My Conditional Stage') {
    when {
        branch 'master'
    }
    steps {
        echo 'Do that only on master branch'
    }
}

For Scripted pipeline ; if value of i is 0 then we need skip the stages ; obviously we can put a conditional logic by surrounding the stage but if we do that the UI doesn't show it like it skipped

A Scripted Pipeline is literally a Groovy script, so you can just use if/else conditionals like you would in a normal Groovy script.

if (env.BRANCH_NAME == 'master') {
  // do steps here
} else {
  // do other steps here
}


Approaching user adds and edits with Ansible I am going to be attached to a small project that basically organizes the users in a cluster of AWS Linux servers. The servers users assigned have different UIDS and maybe different GUIDs. I am hoping to use Ansible to fix this up and organize users better. I can get Ansible to run on a test environment but one thing has got me nervous is that /etc/passwd gets completely screwed up! So my questions are as follows:

Would a better approach be to overwrite /etc/passwd using Ansible. So basically I have a template of /etc/passwd and use a combination of Jinja2 and Ansible vault to rewrite the values in /etc/passwd (I am currently doing this with sudoers and it works well)? If this is the case, would I need to do anything with /etc/shadow or am I safe there?
Is Ansible smart enough where I won't need to use /etc/passwd? For example, I have user bob assigned to UID 5001 when he should be consistently assigned 5002. If I was to readd the user using the users module and specifically assign 5002 to his UID, would that work better?

Please let me know if you need any more information.

If editing critical flat-files across multiple servers in one place is too scary..

One approach I have often recommended is to let system users reside in /etc/passwd files, and to add an additional Name Service Switch (NSS) source for humans that need to log in. For big projects (lots of users authenticating against the PAM user data), that means LDAP, and for small ones (few can be hundreds of such users), nss_db + pam_userdb.

With multiple NSS and PAM sources for users, you can avoid disabling a server when the human user accounts distribution fails for whatever reason. Services (like your privileged Ansible account) should always depend ONLY on the standard /etc/passwd files. LDAP (or even MySQL) is probably overkill for some cases (like yours).

Consider the old BDB backed gems, designed originally (for big FTP file distribution servers) to provide faster logins than /etc files, but otherwise work nearly the same. They store the same passwd and groups flat files' content, but loaded into a (now non GPL) BerkeleyDB NOSQL key/value database file which is faster to read. For this purpose, it's just an extra place to store user/group authentication/authorization data.

NSS

https://en.wikipedia.org/wiki/Name_Service_Switch

https://www.systutorials.com/docs/linux/man/5-nsswitch.conf/

https://sourceforge.net/p/nssdb/home/Home/

PAM

https://en.wikipedia.org/wiki/Linux_PA

https://www.systutorials.com/docs/linux/man/8-pam_userdb/

https://www.cyberciti.biz/tips/centos-redhat-vsftpd-ftp-with-virtual-users.html

steps

collect your passwd, shadow, and group files on a secure server with
the nss_db Makefile
build the nss passwd and group db files using the provided nss_db
Makefile
extract the users and password hashes from shadow file and build PAM db file

awk -F: '{print $1;print$2}' shadow > users.txt # used like the vsftpd vusers.txt example

db_load -T -t hash -f users.txt pam-users.db

distribute the files to somewhere appropriate on your target servers
(maybe /var/db/authdb/*.db)
set up the target servers' pam.conf and nsswitch.conf files to use
these db files
Steps 1-4 are the repeatable build/deploy process.

why?

Once you have working pam/nss configs, your risk of breaking services
by updating/corrupting user auth/auth is very low.
User authentication will work on each server with no external network dependencies like LDAP or MySQL etc.
You can treat the DB files as immutable artifacts on the farm to reduce/prevent user auth snowflakes even if you can't have immutable servers.


How to remotely update AMI id in Jenkins EC2 plugin? I have been looking for a way to update AMI id in Jenkins EC2 plugin configuration after a packer build is run. After some digging, I found a promising way to do it IF the packer run is done by Jenkins itself, via postbuild groovy plugin (mind you, this remains to be tested, but looks good)

However I would like to be able to run the packer build anywhere, and have the resulting AMI id updated at Jenkins remotely, presumably via authenticated REST? Is that possible? Where would I start looking?

The PR you are linking to contains a link to a python script that does exactly this, since you can run that script from anywhere you can access your Jenkins server.


Jenkins: make a credential available to all users I'd like to allow all of our organization's users to create new pipelines in Jenkins, via Blue Ocean...

This requires a Github access token, meaning I have to either require each user to use their own access token, or share the contents of an access token with all users.

The first option seems inconvenient. The second option is a security risk,  and we'd have to change our token any time someone left the organization.

I was hoping to find a way to add a credential to Jenkins and scope it so that all users have access to it, so the above New Pipeline flow would use that cred rather than querying the user to input a token.

I created a token with Global (rather than System) scope, but new users are still asked to input a token when they go to add a pipeline.

Is there a good solution for this? Do most organizations rely on individual developers to use their personal access tokens?

I was a little confused by the OP question and follow up, so answering about GitHub access to jenkinsfile-based pipeline jobs:
Credentials stored in Jenkins can be available to all users with rights to create jobs. Use a GitHub deployment key for your repo. Yes you need a key for every repo. This does require a user in Github, of course, but there is nothing to stop you from paying for an extra so-called 'machine user.' We, however, just use a real person like me.

This GitHub deploy key allows you to trigger on repo changes, or keep your jenkinsfile in Github. Using jenkinsfiles and keeping them in GitHub, is the best way to do Jenkins, I think.


Unable to start step function after account move I recently moved my development account away from my prod one, everything seemed fine except I have a service that runs a lambda to start a step function - not ideal but used to work.

My issue comes in that I am now able to run everything up to and including the execute step function from the npm package provided by aws. I have checked and the account ID is updated in the arn of the step function/machine. I see in my cloudwatch logs that it is timing out after 30 seconds (had a 6 but changed to test).

What are some possible causes of a step function not executing after an account change with no code changes other than the account ID update.

Issue is resolved by updating the subnet to a private subnet which uses a nat gateway instead of a public subnet with an internet gateway. This SO post lead my in the right direction.


Is SSL/HTTPS Needed Between Docker Containers on the Same Network? As the title of the questions asks, is it important/needed to have ssl/https working between docker containers on the same network?
Our setup is an Nginx container that proxies two Tomcat containers.
The Nginx container performs ssl termination for the internet cloud.
Then the http request is forwarded to one of two tomcat instances in the same docker network.
For security/safety should the communication between the Nginx container and the two Tomcat containers happen over ssl?
Thank you for your time.

Generally - no, I don't think you need TLS between your reverse proxy and service containers within same Docker network (or possibly 2 Docker networks: 1 internal for workers, 1 with internet access where reverse proxy sits and bridges the gap into internal network).
A wider answer would primarily depend on whether you trust your Docker host and services on it. Then consider how containers address each other, do they need to leave docker network (like public name resolution) at any time. And what you'd be looking for from TLS: identity verification or transport encryption. Each of those considerations would have some caveats.


Git-based CI/CD Pipeline for Monorepo Are there established CI/CD strategies & best practices for working with a distributed team on a project that is utilizing monorepos?

I can provide two specific use-cases:

Working on content distribution, e.g. a daily/weekly HTML email newsletter, and
A multi-project directory for building LaTeX documents

Both of these scenarios will utilize the same CI/CD pipeline (build/send an email, compile and upload a LaTeX document) but with a growing & variable amount of content that makes independent repos per document undesirable.

Most of the projects using monorepos currently seem to be javascript projects with a lot of cross-dependencies on code. This is a step a way from that, but I'd still like to embrace the audit trail of version control with an automated development process for producing content.

Do these projects compile every piece of code on change? Is there a way to embrace a git diff or use some sort of 'flag' file in the repo root to use CI/CD tools with a monorepo?

The fact that all the code is in a single repository doesn't mean that all the code is changing every time a commit is pushed. I would first make the "pathways" in the code explicit. E.g. perhaps you have a few subdirectories:

- App1/
  - Docs/
  - Code/
  - Tests/
  - Makefile
- App2/
  - Docs/
  - Code/
  - Tests/
  - Makefile

Assuming that App1 is independent from App2, you could trigger it's build by checking if anything in that directory changed (see https://stackoverflow.com/questions/424071/how-to-list-all-the-files-in-a-commit).

Most continuous integration tools have a git plugin or native feature that allows you to filter on a path. If you were using Jenkins, you could set up a few jobs, each with the included region to be be the pathway that should be triggered -- so, e.g. changes to App1/* would trigger jobs/app1/ which executes make -f App1/Makefile or similar.

The trick then to maintaining this project is really in the Makefiles, or other build tool of your choice. You would write the dependencies in there.

You could also have a single global Makefile and use the git diff-tree trick in the question mentioned above:

if grep -q App1 `git diff-tree --no-commit-id --name-only -r <commit>` ; then
  # trigger pipeline App1
  cd App1
  make build
  make test
  make publish
fi


Azure Devops x On-Premises Do I need to allow Azure DevOps access to the On-Premises environment? According to the document at "https://docs.microsoft.com/en-us/azure/devops/organizations/security/allow-list-ip-url? view = azure-devops, " on "Inbound connections" connections are made from the 191.235.226.0/24 (Brazil South) network, but we are receiving connections to our IPs 40.65.126.175, 40.117.36.92 and 52.152.147.38.
And the IPs are changed with each access making it impossible to restrict access.
Know someone who can answer?

What are you trying to achieve?
If your goal is to run build agents to execute pipelines on premises, then you don't have to allow inbound access. Build agents will query Azure DevOps for any new jobs, so the connections are actually outbound.


How to enable tag immutability in quay.io? Elastic Container Registry (ECR) for example supports Tag Immutability (TI). TI implies that docker tags cannot be overwritten if they exist in a docker-registry.

The settings of the organization and the ones from a repository were inspected, but no TI setting was found.

Questions

Does quay support TI?
If true, how to enable it
If false, how do you or does your organization prevent that tag overwrite takes place?

At first glance, it seems that tag immutability is not a feature provided by Quay. You can enable "Trust and Signing" in the "settings" of the repo:

Please note that this will not prevent users from overwriting signed tags without updating signatures. This means that:

Any tag operations in the UI or client can cause inconsistency
Builds should not push to signed tags

We recommend you maintain a strict separation between signed and unsigned tags to avoid any issues with garbage collection.

So, you can sign tags, but you can't prevent them from being updated.


How to make different permissions for different subdirectories of storage bucket On Google Cloud Storage or Amazon S3, is there a way to give users a key to only a specific subdirectory of a storage bucket, for example:

gs://images.eng.cm.com/avatars/<user-uuid>/xyz

so the user has a user id of  and they can read and write only to that subdirectory of the storage bucket, and only if they have a read/write access key in hand too?

Anyone know how to create this fine-grained access control? is it possible/advised?

Before giving the answer I'd like to clarify that S3 is meant for object storage, meaning that there's no file system involved and thus there are not "directories" per se.

So, an "object" with the name "/avatars/user-123/xyz" might give you the impression that it's stored in the "/avatars/user-123" directory, but there's no directory involved, that's just a name.

That being said, you can treat that portion of the name (/avatars/user-123) as a prefix, and then you can create IAM policies for your S3 buckets using said prefixes.

For futre reference, the following examples have been taken from:
https://aws.amazon.com/premiumsupport/knowledge-center/s3-folder-user-access/

In case readers find out that link has been taken down I'll just copy and paste the examples here

AllowStatement1 allows the user to list the buckets that belong to their AWS account. The user needs this permission to be able to navigate to the bucket using the console.

AllowStatement2A allows the user to list the folders within awsexamplebucket, which the user needs to be able to navigate to the folder using the console. The statement also allows the user to search on the prefix media/ using the console.

AllowStatement3 allows the user to list the contents within awsexamplebucket/media.

AllowStatement4A allows the user to download objects (s3:GetObject) from the folder awsexamplebucket/media.

{
 "Version":"2012-10-17",
 "Statement": [
   {
     "Sid": "AllowStatement1",
     "Action": ["s3:ListAllMyBuckets", "s3:GetBucketLocation"],
     "Effect": "Allow",
     "Resource": ["arn:aws:s3:::*"]
   },
  {
     "Sid": "AllowStatement2A",
     "Action": ["s3:ListBucket"],
     "Effect": "Allow",
     "Resource": ["arn:aws:s3:::awsexamplebucket"],
     "Condition":{"StringEquals":{"s3:prefix":["","media"]}}
    },
  {
     "Sid": "AllowStatement3",
     "Action": ["s3:ListBucket"],
     "Effect": "Allow",
     "Resource": ["arn:aws:s3:::awsexamplebucket"],
     "Condition":{"StringLike":{"s3:prefix":["media/*"]}}
    },
   {
     "Sid": "AllowStatement4A",
     "Effect": "Allow",
     "Action": ["s3:GetObject"],
     "Resource": ["arn:aws:s3:::awsexamplebucket/media/*"]
   }
 ]
}


How to install docker in Raspbian Virtual Machine I first installed docker in raspberry pi using: https://iotbytes.wordpress.com/setting-up-docker-on-raspberry-pi-and-running-hello-world-container/
which successfully did.

I tried doing the same with Raspbian VM but using curl -sSL https://get.docker.com | sh says "Unable to locate package docker -ce"
Please help me with the detailed procedure.
I am using Oracle Virtual Box (Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08))

The Docker website doesn't know about your operating system and has no docker installation for you.

Try the one packaged by raspbian:

sudo apt install docker.io

The name is different in some distributions, you can use:

apt-cache search docker

To see if you can find one. If you can't find any, then switch to an older (or newer?) version of Raspbian.

Or you can search for instructions on how to compile it from source code.


py2neo AddressError When Connecting to neo4j in Docker I am having difficulty creating a Graph object on Neo4j 3.4.6 using py2neo 4.10 with Python 3.7.0.

I created a Docker container running Neo4j, and I disabled authentication using Dockerfile entry ENV NEO4J_AUTH=none. I verified that I can browse to the Neo4j database from the host with http://localhost:7474 and that I was not required to enter a password.

I created a second Docker contain for my web server. I accessed the Bash shell using Docker exec -it 033f92b042c1 /bin/bash. I verified that I can ping the Docker image containing the Neo4j installation.

From the second Docker image, I tried to create a Database object or a Graph object.

import neo4j
import py2neo
from py2neo import Graph
graph = Graph("bolt://172.17.0.3:7687")

I tried different protocols and localhost rather than the IP. In each case, Python throws this:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 88, in __new__
    inst = cls._instances[key]
KeyError: '5280a6d494b601f0256493eab3a08e55'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 305, in __new__
    database = Database(uri, **settings)
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 92, in __new__
    from py2neo.internal.http import HTTPDriver, HTTPSDriver
  File "/usr/local/lib/python3.7/site-packages/py2neo/internal/http.py", line 26, in <module>
    from neo4j.addressing import SocketAddress
  File "/usr/local/lib/python3.7/site-packages/neo4j/addressing.py", line 26, in <module>
    from neo4j.exceptions import AddressError
ImportError: cannot import name 'AddressError' from 'neo4j.exceptions' (/usr/local/lib/python3.7/site-packages/neo4j/exceptions.py)

Am I missing a dependency, or is there another way I should be connecting the Docker images?

I had a similar error while using a virtual environment using PyCharm.  In my case, I was using an upgraded version of neo4j-driver in my virtual environment compared to my host environment.  I downgraded neo4j-driver from 1.7 to 1.6.2.

Make sure you have neo4j-driver installed in general.


Jenkins pipeline + Docker integration I have a CI server which has Jenkins installed in the host it self, and I want to run builds and tests inside Docker containers. Until here it works like it should, but I also want to deploy containers using docker-compose and this should be done in the host, but when I run commands inside the Jenkinsfile, they are ran inside the Docker container used for builds and tests.

pipeline {
agent {
    dockerfile {
        filename "Dockerfile"
    }
}

stages {
    stage("Run unit tests") {
        steps {
            # This is ran inside the Docker container, as expected
            sh "cp environments/.env.build .env"
            sh "vendor/bin/phpunit"
        }
    }

    stage('Deploy to test') {
        when {
            branch "develop"
        }

        steps {
            # This is still ran inside the Docker container, not the host
            sh 'docker-compose up -d'
        }
    }

    stage("Archive") {
        steps {
            # This is ran inside the Docker container, as expected
            sh "zip -r dist.zip ."
        }
    }
}

post {
    always {
        archiveArtifacts artifacts: "dist.zip"
        junit "reports/phpunit.report.xml"
    }
}
}

One way to solve this it to use multiple pipelines. The first pipeline runs the test in docker. When that has been completed, one could call another job

https://jenkins.io/doc/pipeline/steps/pipeline-build-step/

https://support.cloudbees.com/hc/en-us/articles/221400287-How-to-pass-parameter-to-downstream-job-in-Pipeline-job-

node {
    paramAValue = "paramAValue"
    paramBValue = "paramBValue"
    build job: 'downstream-freestyle', parameters: [[$class:  'StringParameterValue', name: 'ParamA', value: paramAValue], [$class:
       'StringParameterValue', name: 'ParamB', value: paramBValue]]
}

that runs docker-compose on Jenkins itself.


Stolon vs CrunchyData for HA Postgresql I'm using helm charts with kubernetes and I'm trying to deploy High Availability Postgresql database and found this article mentioning three project that provide HA Postgresql:

1- Stolon

2- CrunchyData

3- Patroni

I didn't find helm chart example for Patroni, but I installed both helm charts for Stolon and Crunchy. However, I wasn't able to find the differences between them except in the structure, both use statefulset from kubernetes in order to deploy master service and a number of replicas(ReadOnly slaves),stolon has this proxy service that points to the master service which is like an extra layer, makes the stolon look a little more complex than crunchy.

But what I really need to know is:

How the replicas will synchronize with the master? synchronous or
asynchronous.
Whats the failover mechanism provided if any?
persistence volumes why is there a volume per pod? How is this
related to the replication?

I ended up using stolon for its automatic leader election and fail over handling.
Crunchy Data offers small building blocks so there is a lot of work to do to put them all together while stolon has a ready helm chart.
But I didn't try patroni since stolon worked just fine.


Cross Region Copy of Aurora Snapshots current_date = str(dt.now().strftime('%Y-%m-%d'))
target_cluster_snapshot_arn= current_date+'_Development_Aurora'

response = TARGET_CLIENT.copy_db_cluster_snapshot(
    SourceDBClusterSnapshotIdentifier=source_cluster_snapshot_arn,
    TargetDBClusterSnapshotIdentifier=target_cluster_snapshot_arn,
    KmsKeyId='arn:aws:kms:us-west-2:xxxxxxx:key/axxxxxx-e326-4df2-8274-73f87ff02f37',
    CopyTags=True,
    Tags=[
        {
            'Key': 'Deletion_Date',
            'Value': (dt.now() + datetime.timedelta(days=30)).strftime('%Y-%m-%d')
        },
    ],
    SourceRegion=SOURCE_REGION
)

While using the above code in Python I am getting the error as

botocore.exceptions.ClientError: An error occurred (InvalidParameterValue) when calling the CopyDBClusterSnapshot operation: Invalid cluster snapshot identifier:  2018-05-22_Development_Aurora

But when i am hardcoding the value in TargerDBClusterSnapshotIdnetifier like below, its working fine

response = TARGET_CLIENT.copy_db_cluster_snapshot(
    SourceDBClusterSnapshotIdentifier=source_cluster_snapshot_arn,
    TargetDBClusterSnapshotIdentifier='PrashastTest',
    KmsKeyId='arn:aws:kms:us-west-2:xxxxxxx:key/xxxxxxb3-e326-4df2-8274-73f87ff02f37',
    CopyTags=True,
    Tags=[
        {
            'Key': 'Deletion_Date',
            'Value': (dt.now() + datetime.timedelta(days=30)).strftime('%Y-%m-%d')
        },
    ],
    SourceRegion=SOURCE_REGION
)

Any pointers why is this happening?

Your proposed snapshot-id is not syntactically valid.  When you tried hard-coding the value, you used a valid identifier.

Underscores are not allowed, and the identifier can't begin with a number.

Here are the constraints on snapshot identifiers, from the API Reference:

  DBClusterSnapshotIdentifier

  The identifier of the DB cluster snapshot. This parameter is stored as a lowercase string.

  Constraints:

  Must contain from 1 to 63 letters, numbers, or hyphens.
  First character must be a letter.
  Cannot end with a hyphen or contain two consecutive hyphens.

  https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_CreateDBClusterSnapshot.html


Scheduling a Docker Container or Installing Scheduler in Container I have an interesting question related to scheduling my Docker Container.

I have a virtual machine with a docker image inside of it. I can spin up a container from the image that has an entrypoint (jar).

I am in a dilemna on how I should schedule the jobs to run against the jar.

Here are my 2 options:

1) If I run a cron on the VM then I can call the docker run commands to produce a container and run different jobs against the jar. The issue here is that once a single job is run, the container automatically terminates and the second job has no previous histroy of the previously run jobs. I believe theres a way to keep a container running forever with an entry point but I haven't explored/tested this yet.

2) I can install a cron inside of the container and through the Dockerfile provide the cron for updating - however, the issue here is how would I launch the container and keep it running for the cron jobs to run?

If you need any further details please do let me know.

Thanks!

You don't need cron. Most server OS-es (CoreOS, RedHat, CentOS, Ubuntu, etc.) these days run systemd, and systemd unit files can run your container to execute the job, for example:

Two unit files. One is mytimedstuff.service and mytimedstuff.timer. The .timer defines when the service should be executed.

In mytimedstuff.service

[Unit]
Description=Executes mystuff
After=default.target docker_network_apps.service
Requires=default.target docker_network_apps.service

[Service]
Type=oneshot
User=root
ExecStartPre=/usr/bin/docker pull mystuff:stable
ExecStartPre=-/bin/bash -c "/usr/bin/docker rm -f mystuffcontainer 2>/dev/null"
ExecStart=/bin/bash -c "/usr/bin/docker run --name mystuffcontainer mystuff:stable mystuff"

In mytimedstuff.timer:

[Unit]
Description=My stuff runs at 00:10 sharp.

[Timer]
OnCalendar=00:10:00

[Install]
WantedBy=multi-user.target


Set environment variables then run script in Jenkins Scripted Pipeline I am new to Jenkins, Groovy and pipelines. I have created a simple pipeline stage like so:

//working build but not setting env variables
node('build-01') {
    stage('Building') {
        echo "[*] Starting build (id: ${env.BUILD_ID}) on ${env.JENKINS_URL}"
        try {
            sh 'ls -l'
            //ls shows the damn file
            sh '. setup-target'
        } catch(all) {
            sh "echo 'Failed to run setup-target script with error: ' ${all}"
        }
    }
}

This works. But I want to modify/add environment variables to the session running this script (this script is a bash file with the correct shebang line on top). So I did:

node('build-01') {
    withEnv(["CMAKE_INSTALL_DIR=${WORKSPACE}", "SDK_INSTALL_DIR=${WORKSPACE}"]){
        stage('Building') {
            echo "[*] Starting build (id: ${env.BUILD_ID}) on ${env.JENKINS_URL}"
            try {
                sh 'ls -l'
                //ls shows the damn file
                sh '. setup-target'
            } catch(all) {
                sh "echo 'Failed to run setup-target script with error: ' ${all}"
            }
        }
    }
}

This errors out with:

    /home/jenkins-sw/ci/workspace/myWorkSpace@tmp/durable-6d30b48d/script.sh: line 1: .: setup-target: file not found

and

    Failed to run setup-target script with error:  hudson.AbortException: script returned exit code 1

But the environment variables are set, I check this by doing a sh 'printenv' right below the ls -l line. Interestingly ls -l does show the script.

What am I missing?

I figured it out. I was cloning directly into the workspace and then setting my environment variables to point to the workspace as well. I modified both those things. I now create a dir in my workspace and clone into it and I also set my environment variables to directories inside my workspace. Like so:

node('build-01') {
    withEnv(["CMAKE_INSTALL_DIR=${WORKSPACE}/cmake_install", "SDK_INSTALL_DIR=${WORKSPACE}/sdk"]){
        stage('Building') {
            echo "[*] Starting build (id: ${env.BUILD_ID}) on ${env.JENKINS_URL}"
            try {
                sh 'ls -l'
                //ls shows the damn file
                dir('path/to/checkout') {
                    sh '. ./setup-target'
                }
            } catch(all) {
                sh "echo 'Failed to run setup-target script with error: ' ${all}"
            }
        }
    }
}

This works.

Note: Since this site is in beta phase and I barely got any views I posted the same question to stackoverflow, here it is for refernce: https://stackoverflow.com/questions/56015463/set-environment-variables-then-run-script-in-jenkins-scripted-pipeline


What are the tools you recommend for deploying / automating / organizing vm in the private cloud I manage a set of virtual servers in a private cloud on vSphere.

In this private cloud we have several sets of servers that host several web applications such as:

-Webpress web sites

-Phone applications in PHP

-Java web applications

-Alfresco

-Redmine

-Nagios

-Jenkins

-Odoo

And these applications are hosted in separate servers with an architecture that mixes several technologies:

-firewall

-reverse proxy / loadbalancer / SSL termination (Haproxy)

-caching servers

-web servers

-LDAP directory server

-Server to centralize the backup and send it to remote NAS

We have a problem of complexity to manage these applications as each application has a VM server that hosts it and another that contains its database and another that saves the data ..etc

What do you recommend to me as tools to make my life easier, on deployment, automation, maintenance, organization of applications, security

And on everything how to organize and save my configurations for each server ... etc

Do you recommend another architecture or solution?

The answer will depend on how much you want to change. I will describe how most larger companies solves this nowadays.

Containerize the apps. Run them in Docker containers. And use a container orchestration tool, like Kubernetes to manage deployment. But to do this your application must probably be redesigned to follow The Twelve factor methodology.

Both Amazon, Azure and Google have Kubernetes cloud services. If you want to continue with private cloud, OpenShift is the most popular Kubernetes-product.


Docker Sign an Image with an existing Certificate With respect to docker's security system.

Is it possible to somehow get images signed by some entity that has trust from a company's CA-cert and then be able to find out who signed the image?

It seems like Docker Trusted Registry (DTR), and Notary in general, have a binary mode of trust: either an image is signed or it is not, but it is not qualified by who performed the signing.

This documentation indicates that it is possible to add collaborators by trusting their key and that it is also possible to revoke access.

Having said that, the answer to:

  either an image is signed or it is not, but it is not qualified by who
  performed the signing

that it is possible to check who signed the image if every collaborator creates their own key and each key is trusted individually.

As notary and dtr were already mentioned, an alternative is Portus. Portus is able to restrict access to docker images, e.g. some users are only allowed to view images, while others may pull them. It is possible to revoke access as well.


Azure DevOps - how to create a 2nd Microsoft-hosted Agent Pool I am using Azure DevOps.

I have 5 Microsoft-hosted agents. And 10 pipelines.
One pipeline in particular can get a bit grabby, so I want to restrict it to use just 2 agents.

But I want my other 9 pipelines to have access to the 3 remaining agents.

I figure Agent Pools is the right way to solve this. By default, Azure DevOps sets us up with 1 single "Azure Pipelines" pool with our self-hosted agents. I would like to split this into 2 pools.

However, when I go to "create new pool" I only get the option to add "self-hosted" agents.

How do I create a new Agent Pool with some of my existing Microsoft-hosted agents?

This is a good question because you would assume you could do what you are after, but as far as I know you can't yet. From the docs on sharing of parallel jobs across projects in a collection

  Currently, there isn't a way to partition or dedicate parallel job capacity to a specific project or agent pool.

I considered trying to do buckets via capabilities on a subset of my agents, and then companion demands in the pipelines, but that will not work either because hosted pipelines don't work with demands.

In my case we had some greedy pipelines that fan out into multiple parallel jobs, that we had to update to run sequentially with dependsOn for each job. I think if your case is relatively simple you could potentially check the queue length or number of running builds in a first stage but that could get complex and might be tough to scale.

Hopefully this feature gets added soon. I will look to see if I can find one to vote up on the developer community forums. I am also interested in other ideas on how to solve for this issue!

Edit found a feature vote on it here


How to develop a docker container for laravel without laradock or docker-compose I am very new to docker. I want to build a laravel development environment with docker. I already tried laradock but it seemed complicated for me. I was wondering whether it is possible to create a laravel development environment without laradock or docker-compose

Yes.  You could create each component of your development environment using containers and manually network them together.  This approach would require you to manually start each container and specify the network and volumes they use at runtime.

However, you are adding un-needed complexity by avoiding docker-compose to network your containers together.  By using a compose file you can start your environment with a single command: docker compose up.  In the approach outlined above, you would need to manually re-create the containers every time you wished to recreate the environment.

Laradock's website has some simple getting started guides that can get you up and running without a full understanding of Docker.  While the learning curve may be steep, it can be beneficial to avoid re-implementing a viable solution.


Unregister a VM within VMware Fusion from command line Is there anyway of unregistering a virtual machine on OSX using the command line with VMware Fusion (so that the VM can then be deleted)?

I've looked around, and found this thread, which suggests

  manually remove the VM from inventory before running the DeleteVM command

But where is this inventory? Or is there a better way of unregistering a machine via the command line?

There doesn't seem to be a "proper" way of unregistering a virtual machine sadly.

I went around this by stopping the virtual machine using vmrun in Fusion, deleting the source folder of the VM, and restarting Fusion so that the application registers that the VM is not there anymore.


Continuous integration using bitbucket pipelines failed for OpenEdx I am trying to set up CI process using bitbucket pipelines for my Openedx site. The script that i am using in my bitbucket-pipelines.yml file is given below. I am trying to just set up the build process on LMS (themes), so that whenever someone make any change in the front end of the site, the build updates paver assets and recompile the assets. The problem is, it is failing on paver update assets giving the error:

  python manage.py lms --settings=devstack_docker print_setting
  STATIC_ROOT 2>/dev/null Build failed running
  pavelib.assets.update_assets: Subprocess return code: 1

Any solution?

image: python:3.5.6
definitions:
  services:
    docker:
      memory: 7168
options:
  size: 2x  # all steps in this repo get 8GB memory

pipelines:
  default:
    - step:
        services:
          - docker
        script:
          # Upgrade Docker Compose to the latest version test
          - python --version
          - export DOCKER_COMPOSE_VERSION=1.13.0
          - curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
          - chmod +x docker-compose
          - mv docker-compose /usr/local/bin

          - export OPENEDX_RELEASE=hawthorn.master
          - git clone https://github.com/sanjaysample/devstack.git
          - cd devstack
          - git checkout open-release/hawthorn.master
          - make dev.checkout
          - make requirements
          - make dev.clone
          - ls
          - make pull
          - make dev.up
          - sleep 60 # LMS needs like 60 seconds to come up
          - docker cp ../metronic edx.devstack.lms:/edx/app/edxapp/edx-platform/themes
          - docker cp ../pavelib edx.devstack.lms:/edx/app/edxapp/edx-platform
          - wget https://raw.githubusercontent.com/sumbul03/edx-theme/master/lms.env.json
          - docker cp lms.env.json edx.devstack.lms:/edx/app/edxapp/lms.env.json
          - rm lms.env.json
          - docker cp edx.devstack.lms:/edx/app/edxapp/lms.env.json .
          - cat lms.env.json
          - docker ps
          - docker-compose restart lms
          - docker-compose exec -T lms bash -c 'source /edx/app/edxapp/edxapp_env && cd /edx/app/edxapp/edx-platform && git init && git remote add origin https://github.com/edx/edx-platform.git && git fetch origin open-release/hawthorn.master && git checkout -f open-release/hawthorn.master && paver install_prereqs && paver update_assets lms --settings=devstack_docker --debug || true'

The author of the Q&A indicated that the issue has been solved since absolute paths were used.


What's the term for moving/promoting code between deployment environments I'm looking for a term that unambiguously describes the process of promoting a "deployment bundle" (for lack of a more accurate/technical term) from one deployment environment to another, from Development towards Production (or whatever environments there might be in between). I am talking specifically about the context of release management, where each environment has its own dedicated purpose (development, integration testing, user acceptance testing, staging, production, etc.), and which succeed each other in a defined, usually the same, order (of importance or quality).

I had previously thought this process was called staging (since I heard the term in this exact context, in multiple projects and teams), before research - especially this discussion: What is, or could be the formal and/or officially established definion of a "stage"? - proved me wrong.

My guess is that I might end up using the term deployment itself, but I'd like to further differentiate between the processes of deploying source code to a server and moving to a different environment in accordance with some specific process (e.g. end-to-end test successful, reviewed by release manager, etc.), especially since there might be a different(ly implemented) process for each.

More broadly speaking, I'd like to clear some ambiguity in DevOps terminology, so please also correct any wrong assumptions I might have made in my question.

I don't think there is a generic term for promoting to production. I can tell you that within the devops community, terms are frequently interchanged. As long as the point being addressed is well thought out and made clear, the point can be understood. Some of the terms that are used when referring to the movement between environments are deploy, propagate, move, promote, and release. I will say though, whenever I hear the term stage, I usually associate it with a pre-production environment. I think most others would assume the same.

Within your organization/groups, I think it is important to try to be consistent with your terminology. This eliminates any confusion and assumptions people can make. An example would be, in my organization, we use the term deploy whenever we are referencing moving between environments except when referring to the production environment which we use the term release. This is to emphasize that it is the final movement in the pipeline/process.