Error from server (Forbidden): error when retrieving current configuration of: Resource: "apps/v1, Resource=deployments", I integrate the existing kubernetes cluster to the gitlab instance (omnibus). I get the below error in the deployment stage of CI/CD pipeline:

Error from server (Forbidden): error when retrieving current configuration of:
 Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
 Name: "test-deployment", Namespace: "default"
 Object: &{map["apiVersion":"apps/v1" "kind":"Deployment" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"test-deployment" "namespace":"default"] "spec":map["replicas":'\x01' "selector":map["matchLabels":map["app":"test"]] "template":map["metadata":map["labels":map["app":"test"]] "spec":map["containers":[map["env":[map["name":"OHH_COMMON_REDEPLOY" "value":"Sun Feb  9 13:55:45 +0330 2020"]] "image":"192.168.10.6:5000/majid/hello-world:v0.01" "name":"test" "ports":[map["containerPort":'P']]]]]]]]}
 from server for: "deployment.yaml": deployments.apps "test-deployment" is forbidden: User "system:serviceaccount:kubetest-2-bina:kubetest-2-bina-service-account" cannot get resource "deployments" in API group "apps" in the namespace "default"
 ERROR: Job failed: exit status 1

There is my .gitlab-ci.yaml file:

deploy:
  image:
    name: lachlanevenson/k8s-kubectl:latest
    entrypoint: ["/bin/sh", "-c"]
  stage: deploy
  environment:
    name: bina
    url: https://192.168.x.x
  only:
    - master
  script:
    - kubectl version
    - sed -ie "s/THIS_WILL_BE_REPLACED/$(date)/g" deployment.yaml
    - kubectl apply -f deployment.yaml --namespace=default

The deployment file look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
spec:
  selector:
    matchLabels:
      app: test
  replicas: 1
  template:
    metadata:
      labels:
        app: test
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - name: test
        image: 192.168.10.6:5000/majid/hello-world:v0.01
        ports:
        - containerPort: 80
        env:
        - name: OHH_COMMON_REDEPLOY
          value: THIS_WILL_BE_REPLACED

I also create ServiceAccount and ClusterRoleBinding:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: gitlab-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: gitlab-admin
  namespace: kube-system

clusterRole definition:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

How can I resolve this issue?

Going from this, your ClusterRole isn't configured to allow access to deployments, and the ClusterRole you've listed isn't properly bound to your service account. You could configure it with something like I did below as a troubleshooting measure/to make sure you're able to properly configure permissions and rule out an issue with the serviceaccount's role bindings.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: gitlab-admin-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: deployment-reader
subjects:
- kind: ServiceAccount
  name: gitlab-admin
  namespace: kube-system

A new ClusterRole

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: deployment-reader
rules:
- apiGroups: ["extensions", "apps"]
  resources: ["deployments"]
  verbs: ["get", "watch", "list"]

Then to check to make sure the service account can properly access the resource in the default namespace you can check with the following command

kubectl get deployments --as system:serviceaccount:kube-system:gitlab-admin -n default


Is it possible to have an aws EKS infrastructure as code? Is it possible to declare "a complete production level EKS with other aws services" using terraform? I know that terraform has an eks module and an aws provider. But I'm not sure if one can create everything (ALB, IAM roles and policies, security groups, ebs, efs, rds, aws networking, ...) using terraform.
I want to learn aws by experimenting. But I cannot afford spending that much money. So I need to be able to destroy everything and recreate the exact infrastructure whenever it's needed.
I've found a really good aws workshop. But like almost all of tutorials, it's based on running "eksctl and aws cli" commands.
Is there a reliable way of declaring everything once and recreate the whole infrastructure declarativly (without touching aws console)?

Is it possible to declare "a complete production level EKS with other aws services" using terraform?

Yes, and that is what many larger organizations do. I have implemented Kubernetes with other AWS resources this way for larger companies e.g. a bank, using Terraform.

I want to learn aws by experimenting. But I cannot afford spending that much money. So I need to be able to destroy everything and recreate the exact infrastructure whenever it's needed.

That should go well. You can use e.g. terraform apply to provision infrastructure and terraform destroy to take it down. I would also recommend to add tests on your infrastructure using e.g. Terratest and perhaps use tfsec in a pipeline so that you are comfortable that new changes does not break your cluster - using Infrastructure as Code the full way.

Is there a reliable way of declaring everything once and recreate the whole infrastructure declaratively (without touching aws console)?

Yes, that is a best practice.


unable to login using docker registry on jenkins I have a Jenkinsfile that I need to login to ECR. this is the relevant section

        script {
          docker.withRegistry('https://01234.dkr.ecr.ap-southeast-1.amazonaws.com/', 'awsId'){
            def image = docker.image("01234.dkr.ecr.ap-southeast-1.amazonaws.com/tas/master-server:${env.BUILD_ID}")
            image.push()
          }

But what gets output in the jenkins build is the strange part

$ docker login -u AKIAZWB**********HET -p ******** https://01234.dkr.ecr.ap-southeast-1.amazonaws.com/
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: login attempt to https://01234.dkr.ecr.ap-southeast-1.amazonaws.com/v2/ failed with status: 401 Unauthorized

I remember if one runs aws ecr get-login the -u is AWS, not the access key of the credentials. So where do I get this wrong?

I think what you are doing wrong is the way you specify the credentialID to be used.

Instead of 'awsId' try to use 'ecr:ap-southeast-1:awsId'

docker.withRegistry('https://01234.dkr.ecr.ap-southeast-1.amazonaws.com/', 'ecr:ap-southeast-1:awsId') {
    def image = docker.image("01234.dkr.ecr.ap-southeast-1.amazonaws.com/tas/master-server:${env.BUILD_ID}")
    image.push()
}

EDIT: Check the plugin doc


Is there an idiomatic way to create reusable Packer templates? I'm creating about 10-12 Packer templates which almost all work the same way. Same builder (Amazon EBS), with some small variations in AMI names, and almost the same provisioner (Ansible Remote) - sometimes with additional variables and sometimes multiple playbooks.

But, much of the Packer templates remain the same.

I know I can use variables to get some of the changeable values into one place in the file. But I'm keen to cut down on all of the copy-pasta in the templates.

I could do this with a wrapper shell script utilising jq or something. But each complexity I add gives something else for others to learn. Try as I might, I can't seem to find a 'blessed' idiomatic way of doing this with Packer.

Is there an idiomatic way of doing this - or at least a generally accepted way?

So far, I've taken to defining the default builders and provisioners we use, with user variables interpolated in where required, eg:

 "type": "amazon-ebs",
 "instance_type": "t2.micro",
 "ami_name": "{{ user `ami-name` }}

I've got files called eg. packer-builders.json and packer-provisioners.json, and then all the separate Packer templates in their own directories.

I've then written a quick Bash script that takes the name of the template one wants to build, finds all the packer-*.json files, and uses jq --slurp to merge them in (eg. jq --slurp '.[0] * .[1] * .[2]' $LIST_OF_FILES) with the supplied template at the end. It saves the output and sends that along to packer build.

As a bonus, I'm also using jsmin before hitting jq so that I can include comments in any of the JSON files.

It works well, but I'm not a massive fan of it because it's custom and it adds an additional barrier for entry for others coming across it (much more to consider than just packer build something.json!

So I'm really keen for additional answers here if there's a better way to do this.


Doing map and filter of a list of dictionaries in ansible Given a list of users:

# Python Syntax
users = [{'name': 'alice', 'sudo': True}, {'name': 'bob'}, {'name': 'charlie'}]

# YAML Syntax
users:
  - name: alice
    sudo: yes
  - name: bob
  - name: charlie

How would I do the following in an ansible playbook:

# Python Syntax
[ dict(user, home="/home/{}/".format(user['name'])) for user in users if not 'sudo' in user]

# more verbose
for user in users:
  if not 'sudo' in user:
    user['home'] = '/home/' + user['name'] + '/'

resulting in:

# Python Syntax
[{'name': 'bob', 'home': '/home/bob/'}, {'name': 'charlie', 'home': '/home/charlie/'}]

# YAML Syntax
users:
  - name: bob
    home: /home/bob/
  - name: charlie
    home: /home/charlie/

Is this even possible/intended use of playbook language?

There's a lot that you can do, but I'm not sure with the existing filters you easily augment a dict.

Creating a new filter to do it was easier.

$ cat filter_plugins/addhomedir.py
class FilterModule:
    @staticmethod
    def add_home(_val):
        return [dict(user, home='/home/{0}/'.format(user['name'])) for user in
                _val if 'sudo' not in user]

    def filters(self):
        return {
            'add_homedir': self.add_home
        }
$ cat test1.yml
---
- hosts: localhost
  vars:
     users:
         - name: alice
           sudo: yes
         - name: bob
         - name: charlie
  tasks:
      - debug:
          msg: "{{ users | add_homedir | list }}"

You can use set_facts and loop to get the values you want, but the question becomes what format and use of the data is really needed.  I'm not sure creating a new structure is the Right Way (TM).  It could be easier to just calculate the individual value for each item in the list when it is used.

$ cat test2.yml
---
- hosts: localhost
  vars:
      users:
        - name: alice
          sudo: yes
        - name: bob
        - name: charlie
  tasks:
     - user:
         name: "{{ item['name'] }}"
         create_home: true
         home: "{{ '/home/{}/'.format(item['name'] }}"
       when: "'sudo' not in item"
       loop: "{{ users }}"


How does using packer and terraform in a CD pipeline compare to docker images built from git? At work we have a Go CD pipeline for producing docker images, and scheduling docker containers with rancher.

This works quite well. You can commit a change to the Docker image in Git, the pipeline will pick it up, and upgrade the container. (The CD pipeline tags the version of the docker image with the build number of the pipeline, and this provides continuity throughout the upgrade.)

We have a similar arrangement with AWS AMIs using Hashicorp Packer and terraform. You can make a change to a packer json file in git, the pipeline will rebuild a new AMI.  Given the user's approval in Go (we have some stage-gates in Go) this can then in theory to stop the existing ec2 instance and start a new one based on the AMI that was built. (The ID of the AMI is passed through the pipeline.)

In practice terraform doesn't work quite as well in a pipeline. (Compared to docker). Sometimes it is just the sheer complexity of things associated with an EC2 instance and corresponding VPC/routing arrangements can tangle it. Sometimes it is that you need to custom-write your own tests to see if the EC2 instance was stopped, and then started.

Is there a broader consensus on this issue? Are there some patterns I'm missing? Or is it just that when you're on the bleeding edge, things hurt a bit?

My question is: How does using packer and terraform in a CD pipeline compare to docker images built from git?

You're comparing apples with oranges according to your description. A strict comparison could be packer toward dockerfile, and/or terraform toward kubernetes yaml for example.

Main difference between your docker pipeline and your packer/terraform one is the architecture behind them.
What is putting the burden here is that your AWS infrastructure doesn't take advantage of auto-scaling groups rolling updates, or your 'stage-gates' in go does disrupt them.

You're not comparing docker with terraform, you're comparing rancher with your AWS infrastructure choices. If you don't use the equivalent bricks of rancher load balancing and deployment groups (kubernetes wording here) which are Elastic Load Balancers and Auto Scaling Groups in AWS you can't compare them.
That's saying a car and an bicycle does the same thing (transportation) with a better result for the car, put a motor on the bicycle and you'll have a far different appreciation and something less biased.

Your observation is pointing out that your usage of AWS facilities is sub optimal, terraform (or whatever provisionning system) can't help you more than running things, if your architecture is not made (in the terraform file) to allow rolling updates from this facility, no automatic system will fix it for you as you'll have to code it again (test instances status, etc.).

Of course rancher pre-bake most of those, making them invisible to you. This also mean you are enforced in rancher methodology (well, swarm, kubernetes or messos architecture to be exact).
That's the main difference, on one side you have an infrastructure already ready to serve with rancher and you can't run outside it, and on the other side you have to think your architecture before hand. If you were to run docker containers on a bare docker host (without any orchestrator on top) you'll end up with the same architecture design to think before hand and the same caveats you're seeing on AWS.

AWS is not strict on how you use it, it's an infrastructure as a service with no opinion on how you use it, whereas rancher enforce a bunch of services around your own container to save you the design time as long as you accept the constraint within it.

Hope this help seeing the problem from an upper level and as such I think the answer to your questions is that you were missing a pattern ;)

For the records: we're doing deployments on AWS with cloudformation (native version of terraform, with its own caveats) from  a push on a gitlab server and it works nicely, the work to get proper cloudformation templates and the script to run it was non negligible. This to say the tools are not the problem, it's all a design choice at a point, same results can be achieved, arncher comes with its own caveats and design choices to make when you integrate it into an existing infrastructure.


how long does it take to deploy a serverless function? using any method - cli, serverless.com, terraform, roughly:

how much time can i expect to pass between these two steps:

submitting a new function to AWS (or azure or GCF), and
having that function hosted live in the cloud?

note that i'm not asking about cold start times.

i'm wondering how long it takes new function code to be entered into and hosted in the AWS lambda system (or azure, ibm, google etc) for a given region.  that is, after taking steps to deploy some new lambda/serverless function code, approximately how soon will i be able to use it?

This will vary based on your cloud provider and what services you are using.  AWS Lambda functions are designed to be highly available for the region.  Per the AWS Lambda FAQ:

  When you update a Lambda function, there will be a brief window of time, typically less than a minute, when requests could be served by either the old or the new version of your function.

So the new function will not be instantaneously available, and there is a possibility that the old function may be called.


Overview of Gitlab CI/CD process for multi-project SOA application. Releasing and Versioning I have a SOA-application. Each service is a separate project in Gitlab with its own Dockerfile. So, in terms of structure it looks as follows

\UserService
    Dockerfile
    \src

\OrderService
    Dockerfile
    \src

There is one developer (or group of developers) responsible for just one Gitlab project (or service as I call it). So, in fact each group technically can tag its project with its own version, so that UserService can have version 1.2.3, while OrderService can have version 5.7.8. It does not look good, because both services are part of one SOA-application.

So, in terms of DevOps I need know what is the best practice for versioning SOA-application containing physically separate services with their own code base?

My another question is about Docker. We are using docker for packing services. I know, that we can tag Docker images also with version number. In terms of best practices should this image version have something to do with versions of the source code of each single service?

Besides, I do not know what should we treat as a release and its version. I guess there will be a build pipeline in Gitlab, which will build multiple Docker images and tag each one with a version. Should we treat the collection of these images as a release or should there be some kind of a bundle, physical release with again its own version.

I know there may exist different practices, I want to know some real world approaches.

Managing versions of different services can be tricky. For now I've never found a good HOW TO out there. Like most things you do what suit you the best. Here is a couple of suggestion you could use:

-> Tag docker image with their commit sha:
Pretty easy to do when you are using gitlab this is an already set environment variable that you can uses when building you image in the CI/CD pipeline. It's easy to track what code this container. Also when rolling back one of your service you can see directly the merge request you are rolling back on.

-> Continuous deploy on your staging environment:
When one is merging a new feature on the master of one of your services deploy it as soon as possible so you can test the impact on all other services.

-> When staging looks good trigger a release job:
That job release shall be present on all your services. The purpose of this job is to:

Check if the service current version (commit sha) match the currently deployed in production.
If not trigger the deploy job to production for this commit sha pipeline.
If the deploy to production job work then we shall store the current sha deployed in a new folder somewhere in a K/V store that way we keep a single source of truth of what's is being deployed and when

More on that K/V store and release job:

This shall be trigger form a different pipeline on a project you use only to remotely trigger the updated versions on your whole SOA-application.
That way we have a repo that contain a script that will trigger the job release of all the services.
It will create a new folder on your K/V store and all the triggered release job of your application will write their current version in it (updated or not)
Once finished only service that had change will be deployed but you will have a major version for your wall application like this on the K/V store:

release_0.1.5:
 - user_service = COMMIT_SHA
 - payment_service = COMMIT_SHA
 - organisation_service = COMMIT_SHA
 - ect..

When this job is replay it will detect that the folder already exist and so will trigger the deploy of all the version present in it so you can rollback your App to an older state.

Deploy to production job on each of your service can be protected to only be triggered from the main release project. But for quick fix reason you could also let it open just make sure it update the current K/V folder.


Jenkins, Jenkins SSH Plug-In, EC2, "ubuntu" user and "www-data" I am trying to install Jenkins to achieve a Github -> EC2 instance folder pipeline. So far everything works well when I target an empty folder... however,  when I target my production folder owned by www-data the SSH copy of the files fail for permission, as Jenkins uses the ubuntu user, the only one I have an SSH login for.

How do I solve this issue?

The ssh plugin gives you both options,
Either send build artifacts to remote server or run a command on the remote server.
The name of the plugin is "send build artifacts over ssh".. use that


trying to display 3 - 5 commands I was trying to display a few commands with  .stdout - the gives me the mixed output also with   "item"
with item it gives me:
just the strings:
uname -a
uptime
but not the output of those commands
this is the play-book with commented stuff there that does not work:
    - name: To Display === UPTIME and KERNEL Version ===
#    - name: "To Display === {{ item }}"
      command: "{{ item }}"
      loop:
        - uname -n
        - uptime -p
        - uname -r

#        command: uname -n
#      shell: uptime -p; uname -r; rpm -qa --last kernel
#      register: OUTPUT
#    - debug: msg="{{ OUTPUT.stdout('\n') }}"
#    - debug: msg="{{OUTPUT.stdout}}"
#    - debug: var="{{OUTPUT.stdout}}"
#      shell: "{{item}}"
#      with_items: ['hostname', 'whoami']
#        - uptime -p
#
#        - uname -r
#        - rpm -qa --last kernel
...

If you just want to print a subset of the results of your commands on your terminal without using one or more -v flags, you can refactor your playbook to show just the standard output from each of the commands using a JSON query. For example:
---
- name: display info
  hosts: localhost
  tasks:
  - name: Display uptime and kernel info
    shell: "{{ item }}"
    with_items:
      - uptime
      - uname -n
      - uname -r
    register: output

  - debug: msg="{{ output.results | json_query('[].stdout[]') }}"

This will yield output similar to the following:
PLAY [display info] **********************************************************************************

TASK [Gathering Facts] *******************************************************************************
ok: [localhost]

TASK [Display uptime and kernel info] ****************************************************************
changed: [localhost] => (item=uptime)
changed: [localhost] => (item=uname -n)
changed: [localhost] => (item=uname -r)

TASK [debug] *****************************************************************************************
ok: [localhost] => {
    "msg": [
        "22:48  up 4 days,  3:41, 2 users, load averages: 1.26 1.39 1.42",
        "localhost.local",
        "20.3.0"
    ]
}

PLAY RECAP *******************************************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


Can we do zero-downtime kubernetes deployment with a single pod? I've inherited an app with K8s services with the following config
spec:
  ...
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate

For most of these light-weight services, we run a single instance of a pod on GCP.  We're expanding into blue-green deploys for some services, but the majority are still run this way.
When we update the GCR image to a new version, the service becomes temporarily unavailable (maybe 10-30 seconds).  I'm assuming this is because K8s is killing the old pod while it brings up the new pod.  Is there anyway to configure K8s to actually bring up a new pod, direct new traffic to it, and only then shutdown the old pod?  This would seemingly allow for a deployment with no downtime, and not rely on having two pods running all the time (saving cost).  All docs I can find typically start with N pods so they don't have this issue...

UPDATE
So the issue is not related to maxsurge (changing it had no effect).  Actually I realized what's happening is that the pod is getting traffic routed to it before the app is actually completely ready.  These are spring apps, so they take like 30 seconds to start, and the service disruption we're seeing is occurring in the 30s window while the app is booting and the pod has already sent traffic to it.  So I guess what I really am after is as follows
how can k8s be configured to bring up a second pod, wait until it passes startup probe/healthcheck, and only then have the first pod route traffic to it and self-terminate?

Is there anyway to configure K8s to actually bring up a new pod, direct new traffic to it, and only then shutdown the old pod?

This is the default behavior. MaxSurge is number of additional pods that runs during upgrade.
What you also have to check is how the load balancer is handling traffic, does it have capability to detect that you have deployed a new Pod, e.g. is it Kubernetes aware? See Container native load balancing on Google Cloud, it looks like you should have a ReadinessProbe on your Deployment.

Actually I realized what's happening is that the pod is getting traffic routed to it before the app is actually completely ready.

This is the default behavior, if you define a ReadinessProbe for your Deployment. As you write, you can also add a StartupProbe but the ReadinessProbe is most important.


How to enable debug mode for Docker client on Windows 10? By default, Docker client is not in debug mode. How to enable it?

   > docker info

    ..
    Debug Mode (client): false
    Debug Mode (server): true
    ..

If I try following docker --help

docker -D version

I get indeed

Debug Mode (client): true
Debug Mode (server): true

But then, how can I access log messages? The following for example does not seem to bring any debug statements to the console:

PS U:\> docker -D -l debug pull ubuntu:16.04
16.04: Pulling from library/ubuntu
75c416ea735c: Pull complete
c6ff40b6d658: Pull complete
a7050fc1f338: Pull complete
f0ffb5cf6ba9: Pull complete
be232718519c: Pull complete
Digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f4
Status: Downloaded newer image for ubuntu:16.04

Eventually, using the docker -D -l debug setting for the client  I have found the log outputs (apparently currently there are no debug level messages implemented as reported.

C:\Users\user\AppData\Local\Docker\log.txt


Terraform: apply only one tf file I have my security groups in a securitygroup.tf file.

In the same dir there are plenty of other resource descriptions (rds, ec2 etc).

Is there a way to perform a terraform apply --auto-approve only for my securitygroups.tf?

Not really. The standard way to work around this though is to use eg:

terraform apply -target=aws_security_group.my_sg

but that's only going to apply one security group at a time, so will get tedious if you have a lot of them. You can, however, target multiple resources in one command:

terraform apply -target=aws_security_group.my_sg -target=aws_security_group.my_2nd_sg

However, there are potentially a couple of workarounds:

The -target parameter respects dependencies.

This means if you were to eg. -target=aws_instance.my_server and that instance had, say, five security groups attached to it via interpolation, changes to those security groups should be included in the plan (I haven't thoroughly tested this, but I believe this is how it works).

That is a bit messy though, as you probably don't want to touch an instance. A safer alternative might be using something like a null_resource to provide a target for the security groups, but again I haven't tried this (there might be another 'safe' resource you could rely on, though?).

Create a module.

You can target a module just like you can target a plain resource:

terraform apply -target=module.my_security_groups

Inside this module, you could define all of your security groups - just like you would have outside of the module. As well as being able to target it directly, this also makes it easier for you to re-use the same set of security groups for other infrastructure, if you ever need to.


Strategies to work around 30 sec timeout I received a timeout when trying to perform 7000 * 3 inserts into MariaDB in 30 seconds with AWS Python Lambda. I have three ideas how to get around the 30 second API Gateway timeout:

Convert the data to a file which MariaDB can import using LOAD_DATA_INFILE and let the Lambda do that.
Write the data in a file to S3 and let another function write the data from S3 to MariaDB which then will not have the API Gateway timeout limit.
Let the web client convert the file in JavaScript.

I am leaning towards the first strategy, to try and use LOAD_DATA_INFILE from the Lambda function. Do you agree?

A variation of method 2 is infinitely scalable:

write the data to s3.
from s3 trigger lambda to break s3 file into sqs queue events of good size
from sqs queue trigger lambdas to insert
sqs can be fifo queue if it matters.


Create multiple EC2 instances using python script (in loop iteration) I Have JSON file in which email address and count of instances are specified.I'm using python script to iterate through that file and get username/count number (count number will be used as number of EC2 instances), output from JSON file:

DJukes
1
JWilson
2
eflame
3

So, for DJukes one instance will be created,JWilson 2,eFlame 3, also these names will be usernames for new instances.

Idea is to create as many instances as it's specified in JSON file based on username and count values

When creating manually i got

Terraform will perform the following actions:

-/+ aws_instance.win-example (new resource required)
      id:

                   "i-073bda5b30f10d3ba" => <computed> (forces new resource)

I passed these variables to python-terraform wrapper (https://github.com/beelit94/python-terraform)

If only one user/count exists in JSON file, no issues, but as soon i add more entries in JSON, then machine for DJukes is created then terminated,same for JWilson, only last created machine remains running.
Can someone point out what's wrong with this code ?

#!/bin/python
import json
from pprint import pprint
from python_terraform import *

def myfunc():

  tf = Terraform(working_dir='/home/ja/terraform-course/demo-2b', variables={'count':count,'INSTANCE_USERNAME':user})
  tf.plan(no_color=IsFlagged, refresh=False, capture_output=False)
  approve = {"auto-approve": True}
  print(tf.plan())
  print(tf.apply(**approve))
  return

    json_data=open('./my.json')
    data = json.load(json_data)

    json_data.close()

    for i in range (0, len (data['customers'])):
        #print data['customers'][i]['email']
        k=data['customers'][i]['email']
        #print(k.split('@')[0])
        user=k.split('@')[0]
        #print(user)
        count=data['customers'][i]['instances']
        #print(count)
        #enter = int(input('Enter number of instances: '))
        myfunc()

windows.tf:

resource "aws_key_pair" "mykey" {
  key_name = "mykey"
  public_key = "${file("${var.PATH_TO_PUBLIC_KEY}")}"
}

resource "aws_instance" "win-example" {
  ami = "${lookup(var.WIN_AMIS, var.AWS_REGION)}"
  instance_type = "t2.medium"
  count="${var.count}"

  key_name = "${aws_key_pair.mykey.key_name}"
  user_data = <<EOF
<powershell>
net user ${var.INSTANCE_USERNAME} '${var.INSTANCE_PASSWORD}' /add /y
net localgroup administrators ${var.INSTANCE_USERNAME} /add

winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="300"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'

netsh advfirewall firewall add rule name="WinRM 5985" protocol=TCP dir=in localport=5985 action=allow
netsh advfirewall firewall add rule name="WinRM 5986" protocol=TCP dir=in localport=5986 action=allow

net stop winrm
sc.exe config winrm start=auto
net start winrm
</powershell>
EOF

  provisioner "file" {
    source = "test.txt"
    destination = "C:/test.txt"
  }
  connection {
    type = "winrm"
    timeout = "10m"
    user = "${var.INSTANCE_USERNAME}"
    password = "${var.INSTANCE_PASSWORD}"
  }

tags {
Name="${var.INSTANCE_USERNAME}"
}
}

The main issue I see is that you are calling Terraform at the end of every python loop. This is creating new "state" file. That will make it really hard to manage what you are creating. This is how I would do it.

users = []
count = 0
for customer in data['customers']:
    username = customer['email'].split('@')[0]
    instances = customer['instances']
    count += instances
    users.extend([username] * instances)
    #print(Added {} to build list with {} instances.".format(username, instances))
myfunc(count, users)

then I would change...

def myfunc(count, users):

  tf = Terraform(working_dir='/home/ja/terraform-course/demo-2b', variables={'count':count,'INSTANCE_USERS':users})
  tf.plan(no_color=IsFlagged, refresh=False, capture_output=False)
  approve = {"auto-approve": True}
  print(tf.plan())
  print(tf.apply(**approve))
  return

and finally the Terraform.

resource "aws_key_pair" "mykey" {
  key_name = "mykey"
  public_key = "${file("${var.PATH_TO_PUBLIC_KEY}")}"
}

resource "aws_instance" "win-example" {
  ami = "${lookup(var.WIN_AMIS, var.AWS_REGION)}"
  instance_type = "t2.medium"
  count="${var.count}"

  key_name = "${aws_key_pair.mykey.key_name}"
  user_data = <<EOF
<powershell>
net user ${var.INSTANCE_USERS[count.index]} '${var.INSTANCE_PASSWORD}' /add /y
net localgroup administrators ${var.INSTANCE_USERS[count.index]} /add

winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="300"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'

netsh advfirewall firewall add rule name="WinRM 5985" protocol=TCP dir=in localport=5985 action=allow
netsh advfirewall firewall add rule name="WinRM 5986" protocol=TCP dir=in localport=5986 action=allow

net stop winrm
sc.exe config winrm start=auto
net start winrm
</powershell>
EOF

  provisioner "file" {
    source = "test.txt"
    destination = "C:/test.txt"
  }
  connection {
    type = "winrm"
    timeout = "10m"
    user = "${var.INSTANCE_USERNAME[count.index]}"
    password = "${var.INSTANCE_PASSWORD}"
  }

tags {
Name="${var.INSTANCE_USERS[count.index]}"
}
}

What I did was change your code so we now create a list of the users, that match the length of the count. That way we can call Terraform just once, and use the count.index on our users list to make sure we create the correct amount of instances per user.

You can read more about count.index here. I have always found reading and understanding more complicated code makes me better. Try reading some of the modules from the Terraform Registry. The vpc module gives good examples of using count.

If you still need help drop me a comment.


What's in THIS environment? We work on multiple java web projects going on at the same time and those are all being tested on several different QA environments.  I'd like to show on our support portal what projects are in what environments without having to manually update it all the time.  I was thinking of some way to tag the builds so I could just query tomcat or those linux servers themselves and show the results.  The tags would be something like "August Release", "Project 1", "Project 2", etc.. Has anyone done something similar?  I'm looking for different options.

I've done this before, normally in the form of having a /.well-known/info endpoint available on every microservice that then returns a JSON payload with what's in it:
{
    "service_name": "token_exchange",
    "server_id": "RD1717628373",
    "commit_hash": "f72a28",
    "branch": "feature/content-security-policy",
    "version": "v2.8.12-f72a28-prerelease1",
    "dependencies": [
        {
            "id": "db",
            "type": "database",
            "endpoint": "db.mysite.com"
        },
        {
            "id": "service",
            "type": "microservice",
            "endpoint": "RD661872834"
        }
    ]
}

I then have a cron job and custom dashboard that queries all of the services that it knows about and then walks the dependency tree to add any newly added dependencies.
Considerations

Agree a Standard - The first step is to agree on a standard across services that you want to report on, then get them implemented.  It's hard to show the value of this kind of thing when only 1% of your estate implements it.
Start Small - I created a command-line interface that went and queried it at first.
Security - It could be an attack surface from an information security perspective, so test it and possibly add some authentication component to it and turn it off in production.


Chef14 Load a provider in a recipe Chef::Provider::DeployKeyGithub I have a problem during migration of chef-client 12 to chef-client 14 and seems that the old way to load Providers is different in chef14

Chef::Provider::DeployKeyGithub cannot be load

    deploy_key deploy_key_name do
        provider Chef::Provider::DeployKeyGithub
        path "/home/#{current_user}/.ssh"
        credentials({
            :token => File.read('/root/someid').strip
        })
        repo 'test/test_config'
        owner 'root'
        group 'root'
        mode '0400'
        action :add
    end

Error:

================================================================================

NameError
---------
uninitialized constant Chef::Provider::DeployKeyGithub

Cookbook Trace:
---------------
  /var/chef/cache/cookbooks/flume_wrapper/recipes/default.rb:89:in `block in from_file'
  /var/chef/cache/cookbooks/flume_wrapper/recipes/default.rb:88:in `from_file'```

Does anyone knows how is the new way to load a Provider in a recipe for chef14 client?

This compatibility issue was solved for chef14 with the new load format of a provider in a recipe.

deploy_key deploy_key_name do
      provider Chef::ProviderResolver.new(node, find_resource!("deploy_key[github]"), :nothing).resolve
      path "/home/#{current_user}/.ssh"
      credentials({
          :token => File.read('/root/someid').strip
      })
      repo 'test/test_config'
      owner 'root'
      group 'root'
      mode '0400'
      action :add
  end


How to get the ip range of github-production-release-asset-2e65be.s3.amazonaws.com? I have denied all outgoing traffic, but need to allow some of them in order to allow the installation of certain software packages. One of such packages resides on github and that means that the ip range of github-production-release-asset-2e65be.s3.amazonaws.com has to be allowed, but after adding a couple of ip addresses it turns out that there are multiple ips.

Attempt to solve the issue

https://api.github.com/meta was found and returns the ips, but not those of AWS

Current approach

As a workaround I have allow outgoing connection to 52.216/16, but that it is too open

@Tensibai indicated that:

  See: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html if you
  really want to go this way, but there's quite large ranges, the
  52.216.0.0 is a /15 for us-east-1 and not a /16 btw { "ip_prefix": "52.216.0.0/15", "region": "us-east-1", "service": "S3" },


Spinnaker keeps showing disabled stages in Jenkins pipeline UI I have a pipeline with a few disabled stages:
{
  "enabled": false,
  "completeOtherBranchesThenFail": true,
  "continuePipeline": false,
  "failPipeline": false,
  "parameters": {
    "FOO": "bar"
  },
  "type": "jenkins"
}

The doc states that disabled stages are hidden from the UI [1]:

enabled - Determines if stage will be available in the UI.

However, those stages are still present in the UI.
When I go to Configure Pipeline > Edit stage as JSON, the enabled flag is set to false, so the UI has the right value.

https://spinnaker.io/guides/operator/custom-job-stages/#creating-a-custom-job-stage

Spinnaker version 1.23.5

I was not able to find how to hide a stage, however I found one can skip it using stageEnabled [1].
It takes a Spring Language Expression like so:
{
  "stageEnabled": {
    "expression": "false", // you can also use SpEL, e.g. ${someBoolean}
    "type": "expression"
  },
  "completeOtherBranchesThenFail": true,
  "continuePipeline": false,
  "failPipeline": false,
  "parameters": {
    "FOO": "bar"
  },
  "type": "jenkins"
}

Or from the UI editor: Execution Options > Conditional on Expression > Enter false [2].

https://docs.armory.io/docs/spinnaker-user-guides/using-dinghy/#stage-fields
https://spinnaker.io/guides/user/pipeline/expressions/#dynamically-skip-a-stage


Ansible: how to display the name of the currently running included_tasks playbook file? I wrote an Ansible role which does lots of things, it's called "common" and all Ansible hosts are using it.

The directory structure is as follows:

./common
│   ├── ./common/README.md
│   ├── ./common/defaults
│   │   └── ./common/defaults/main.yml
│   ├── ./common/handlers
│   │   └── ./common/handlers/main.yml
│   ├── ./common/meta
│   │   └── ./common/meta/main.yml
│   ├── ./common/tasks
│   │   ├── ./common/tasks/base_packages.yml
│   │   ├── ./common/tasks/iptables.yml
│   │   ├── ./common/tasks/locale.yml
│   │   ├── ./common/tasks/main.yml
│   │   ├── ./common/tasks/pip.yml
│   │   ├── ./common/tasks/pip_packages.yml
│   │   ├── ./common/tasks/python.yml
│   │   ├── ./common/tasks/python2_centos6.yml
│   │   ├── ./common/tasks/python3_centos6.yml
│   │   ├── ./common/tasks/repo.yml
│   │   ├── ./common/tasks/selinux.yml
│   │   ├── ./common/tasks/time.yml
│   │   ├── ./common/tasks/vim.yml
│   │   └── ./common/tasks/yum.yml
│   ├── ./common/templates
│   │   ├── ./common/templates/chrony.conf.j2
│   │   └── ./common/templates/ntp.conf.j2
│   ├── ./common/tests
│   │   ├── ./common/tests/inventory
│   │   └── ./common/tests/test.yml
│   └── ./common/vars
│       └── ./common/vars/main.yml

main.yml looks like so:

---
# tasks file for common
- name: python installation on centos 6 machines
  block:
  - import_tasks: python2_centos6.yml
  - import_tasks: python3_centos6.yml
  when: ansible_distribution_major_version == "6"
- import_tasks: python.yml
  when: ansible_distribution_major_version == "7"
- import_tasks: pip.yml
- import_tasks: pip_packages.yml
- import_tasks: repo.yml
- import_tasks: yum.yml
- import_tasks: selinux.yml
- import_tasks: base_packages.yml
- import_tasks: iptables.yml
  when: ansible_distribution_major_version == "7"
#- import_tasks: time.yml
- import_tasks: locale.yml
- import_tasks: vim.yml

The playbook that I'm currently running:

---

# local_slave_playbook_c6.yml
- hosts: localhost
  vars:
    ansible_user_shell: /bin/bash
  gather_facts: true
  roles:
    - common
    - consul-client
    - hosts
    #- node-exporter
    - cron
    - jenkins-slave

While I run the "common" role on a server, I want to know which imported_tasks file is currently running for debug matters. So for example, while Ansible "takes care" for the installation of base packages, I want to add "base_packages.yml" to the title of the task.

I looked through the list of Ansible special variables but couldn't find the variable that I need.

There is one special variable that looks right "{{ ansible_play_name }}" but that gives me "localhost".

If you want to "add base_packages.yml to the title of the task" you'll have to add it to the names of the tasks. For example

shell> cat base_packages.yml
---
- name: "base_packages: .... "

sed might help to automate it.


In a multistage Docker build, should the WORKDIR be deleted? My Dockerfile contains this:
### STAGE 1: Build Angular app ###
FROM node:14.15 AS build
WORKDIR /usr/src/app
COPY src/hub/package.json ./
RUN npm cache verify
RUN npm install
COPY src/hub/ .
RUN npm run-script build-azdev
RUN npm prune --production

### STAGE 2: Run Angular app in nginx ###
FROM nginx:1.19-alpine
COPY /scripts/nginx/nginx.conf /etc/nginx/nginx.conf
RUN rm -rf /usr/share/nginx/html/*
COPY --from=build /usr/src/app/dist/* /usr/share/nginx/html/
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Once COPY --from=build /usr/src/app/dist/* /usr/share/nginx/html/ is done, should the contents of WORKDIR (/usr/src/app) be deleted once Stage 2 is done building?

When creating an image using a Dockerfile, docker layers are created for each step. The description of layers can be found on the official docker page.
Accordingly, when using Multistage to create an image, it introduces a new build step that can be named using an alias in the FROM statement, and all layers except the image layer created via the last FROM statement are deleted.
After build your image, check if the layer that contains WORKDIR whether exists or not using docker image inspect. I think you probably know the unique layer value for each step.
You can see the image consists with which layers.
e.g.,
$docker image inspect [OPTIONS] 464503b73bfe # IMAGE
[
    {
        "Id": "sha256:464bf1...46c",
        ...

        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:26711...8fcf",
                "sha256:6c5a51...56e0"
            ]
        },
        ...


Should my zombie backend container be restarted when database is down? I have a situation where I have a Postgres and a Node server which connects to it. I am using Kubernetes to deploy them.

If my Postgres container fails and stops working then its livenessProbe will restart him which will take some time, meanwhile, the server container is useless so the Server's readinessProbe should fail and Ingress stops routing traffic to it. If this happens what should be the behavior of the Server?

Some solutions I thought but was not sure about them:

Server's livenessProbe sees database down and restarts the container (doesn't make sense to me because it could generate a cascade effect, Postgres failure makes the Server's container restart)
Server keep trying to reconnect forever and becomes a useless running container while Postgres doesn't come back
Server tries to reconnect a limited number of times with exponential interval between them, after that it restarts

The standard option is #2, keep retrying the connection, possibly with slower intervals, but I'd have an upper limit to the longest interval. The liveness probe should only fail if that specific container is broken, not any of it's dependencies. If there are cases where your application is failing and won't self recover when the DB is restored, then the liveness probe should fail in that scenario, and I'd consider an init container to check for the availability of the DB before starting the full application.

There should be some logging and metrics enabled to make the DB failure visible so that you debug the correct thing. If there is a concern of too many retries impacting the network, then a service mesh with circuit breaker functionality may be a good option.


How to get logged in username of Jenkins in parameter's Groovy script? In Jenkins parameter, I'm writing Groovy script and in that script I need to pass Jenkins logged in username NOT user. User and username, could be different

Ex - User, that is displayed left to 'log out', could be - Bob Gill and username, used to login - could be - bob

User can be retrieved using User.current(). Please tell me how to get username.

NOTE I don't need username in pipeline code. I need it in Groovy script of parameter.

From the Java API documentation for hudson.model.User:

  String  getDisplayName()
  Returns the user name.

  String  getFullName()
  Gets the human readable name of this user.

  String  getId()

It's not clear to me from your question what data you want (I think it's the user ID?), so I'm not sure which of these methods will get you that data (getId()?).  But in any case, you would call the method like:

User.current().getId()


How to create custom AWS CloudWatch metric for i-node I have created custom CloudWatch alarm for Disk Utilization for root and mount volume to monitor the disk space for my Ubuntu (16.04) instances, but quite often it happens that disk is only 60% full (df -h) and still it's unable to create any more file as the i-node (df -i) already reached its limit. To keep an eye on this, I want to introduce a custom metric for i-node utilization and then want to configure an alarm on it.

Please provide a solution, considering I am relatively new in AWS.

AWS have published a plugin for collectd which allows to send metrics collected using collectd to CloudWatch.

https://github.com/awslabs/collectd-cloudwatch

The collectd monitoring tool runs on the server and can be configured to collect various metrics, among them metrics on disks and filesystems using their DF plugin. https://collectd.org/wiki/index.php/Plugin:DF

You can turn the ReportInodes option of the CollectD DF plugin and then have these metrics reported to CloudWatch as custom metrics using the AWS CollectD CloudWatch plugin.


Azure DevOps: How to pass a parameter from a build to a release? I have an azure devops build pipeline that builds some docker images and a release pipeline that deploys the docker images using some artifacts I publish in the build.
In the build I have some parameters that allow the user to change which docker images are built. Defined like this:
parameters:
  - name: docker1
    displayName: Build docker image 1?
    type: boolean
    default: true
  - name: docker2
    displayName: Build docker image 2?
    type: boolean
    default: true

I now want to make jobs in my release stages dependent on these variables as well.
So something like this:

Now the config in this picture doesn't work.
My question is how does the syntax for this work if this is just a problem of syntax or how do I make these parameters available to the release if this is more of an issue of variable scope?
What I've tried:

Just accessing the parameters with the syntax shown above eq('${{ parameters.docker1 }}', true)
Specifying the Build like one would with predefined variables of the build scope eq('${{ Build.parameters.docker1 }}', true)
looked at variable groups vaguely based on this post Azure DevOps Build and Release pipeline artifact name variable . Variable groups seem to be immutable from the build pipeline and also shared between multiple potentially simultaneous builds so this doesn't seem to work for this usecase.

I can image that one way to solve this would probably be to write all my parameters to a file, upload this as a pipeline artifact and then write variables from this imported artifact in the release stage. I want to avoid this if possible since it feels like a very clunky workaround.

In case anyone ever finds this question: I didn't find a satisfying answer. My workaround was to write my variables to a text file in the build and then publish it, download it in the deployment and publish the values as output variables to a bash task. Looks something like this:
source file_containing_your_vars.txt
echo "##vso[task.setvariable variable=var1;isoutput=true]$var1"

the output variables parameters of this bash would need some reference name, for example job_out and you would then be able to reference the variables from other jobs in that stage like this:
variables['job_out.var1']

If you are running a yaml deployment pipeline (instead of one made in the portal) you can even use job output variable between stages.
For portal pipelines that want to share variables between stages this exists and seems to have helped some people but I didn't get it to work:
https://stackoverflow.com/questions/53042605/how-to-modify-azure-devops-release-definition-variable-from-a-release-task/53060141#53060141


Terraform plan -var-file="myvars.vars" does not run I try to lean terraform for usage with our AWS environment.
I have created a .tf file and terraform init was successful.

In the .tf file I have a number of variables that are populated via .var files (not .tfvar) because I want to assign values per instance/project etc.

The problem is that when I run the command

terraform plan -var-file="myvars.var"

all I get is the terraform usage message.

Where is my mistake?

You define variables in variables.tf file. However, you can define inputs to these defined variables in a .tfvars(recommended) or any other files.

  Values for the input variables of a root module can be gathered in variable definition files and passed together using the -var-file=FILE option.

  For all files which match terraform.tfvars or *.auto.tfvars present in the current directory, Terraform automatically loads them to populate variables. If the file is located somewhere else, you can pass the path to the file using the -var-file flag. It is recommended to name such files with names ending in .tfvars.
  more...

Make sure you are running terraform plan from the directory where the variable file resides.


How to create a docker registry secret without passing the password on the command line? According to this document

https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

one could create a docker registry by running:

kubectl create secret docker-registry regcred \
--docker-server=<your-registry-server> \
--docker-username=<your-name> \
--docker-password=<your-pword> --docker-email=<your-email>

How to create a secret without entering the password as plain text?

It is possible to create a secret without passing on the commandline by reading it from file:

$ kubectl create secret generic ssh-key-secret \
  --from-file=ssh-privatekey=/path/to/.ssh/id_rsa \
  --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub

See https://kubernetes.io/docs/concepts/configuration/secret/ for more information.


Can I check whether ansible is on last batch of rolling update (serial: 25%)? I'm using Ansible to update a pool of EC2 servers with a rolling update, in batches of 25%. In between batches, I'm pausing long enough for the server health checks to start to pass again (since they'll fail while the application server restarts).

The question is, can I skip the delay on the final pass so the playbook will finish sooner? Here is the script. The last line is the API I want:

---
- hosts: tag_deploy_environment_development
  serial: "25%"
  remote_user: ec2-user
  become: no
  vars:
    health_check_healthy_threshold: 2
    health_check_interval: 10
    server_port: 80
  tasks:
    #- debug: var=vars
    #- debug: var=ansible_facts

    - name: "Copy the current server version from disk"
      include_role:
        name: deploy-code
        tasks_from: push-files

    - name: "Wait for {{ server_port }} to become available"
      wait_for:
        port: "{{ server_port }}"
        timeout: 60
        state: started

    # I would like to check whether we are on the final batch,
    # and don't pause in that case:
    - pause:
        seconds: "{{ health_check_healthy_threshold * health_check_interval }}"
        prompt: Waiting to allow AWS health checks to pass
      when: ansible_facts['remaining_batches'] > 0

To solve this, I printed vars and ansible_facts, and eventually stumbled on the ansible_play_batch variable, which is a list of hostnames in the current batch. ansible_play_hosts_all gives the full list of hostnames. (Do not use ansible_play_hosts for this purpose, because as of Ansible 2.7, it works but the docs say it gives something completely different.)

The question: Is this the final batch?
Rephrasing based on the data: Is this batch == the last part of the host list?
Or to make it simpler: Is the end of this batch == the end of the host list?

In Ansible, the check for that is:

tasks:
  - pause:
      seconds: "{{ health_check_healthy_threshold * health_check_interval }}"
      prompt: Waiting to allow AWS health checks to pass
    when: ansible_play_batch[-1] != ansible_play_hosts_all[-1]

  - debug: msg="This is the last batch."
    when: ansible_play_batch[-1] == ansible_play_hosts_all[-1]

Side note: I used the following playbook for testing, to simulate having many/few hosts without bothering to set them up. I couldn't see how to specify the hosts as a list in the playbook, so I added them to a group, then executed the group:

---
- hosts: localhost
  connection: local
  tasks:
    - add_host: { name: host1, groups: test_hosts }
    - add_host: { name: host2, groups: test_hosts }
- hosts: test_hosts
  connection: local
  serial: "25%"
  tasks:
    - debug: var=ansible_play_batch


Deploy dist file in docker file How to write a docker file to deploy a dist file of an Angular Application  In a server.
 I see most of them have used nginx as a server. Why is it so? Is there any specific reason

I want it to be deployed in Apache using alpine as a base image

There are lots of good tutorials for NGINX and it is an easy server to setup with the Angular router.  You can still accomplish this with an Apache server, but you may need more steps in your Dockerfile. As a starting place check out the answer on Stack Overflow Deploy Angular 2 to Apache Server.  These types of server modifications will need to be baked into your Dockerfile.


How do a flaten a list mixed with nested dict in ansible playbook? I have a variable named packages as showed below, it is not a plain list, I have some nested dict in it like browsers.

---
packages:
  - apt-file
  - axel
  - ibus-rime
  - rofi
  - i3
  - ffmpeg
  - xclip
  - cargo
  - httpie
  - ttf-ubuntu-font-family
  - tldr-py
  - glances
  - browsers:
    - brave-browser
    - chromium
  - editors:
    - emacs

I'd like to use instructions like below

- name: Install a list of packages
    become: yes
    apt:
        name: "{{ packages }}"

to install all the pacakges defined in the variable, but I do not know what filter I should use the convert the variable to plain list, I need the items in the nested dictionary.

any help will be appreciated.

I probably should use comment instead group the value in a dict who's value is a list

like

packages:
  - apt-file
  - axel
  - ibus-rime
  - rofi
  - i3
  - ffmpeg
  - xclip
  - cargo
  - httpie
  - ttf-ubuntu-font-family
  - tldr-py
  - glances
  # browsers:
  - brave-browser
  - chromium
  # editors
  - emacs


Gather resource usage by process in a kubernetes cluster I am searching for a tool similar to Prometheus + Grafana that gather and record resource usage especially memory usage by process-ID or process-name.

We have two components that are running different processes and they have memory leak and I want to find which process is leaking.

This is from Weave Scope and it shows all the processes of each pod and their resource usage but it is just live, I want something similar but storing it over time like a Prometheus graph.

There are several alternatives to Prometheus for monitoring your cluster.

While some like Heapster have been superseded by Prometheus, other solutions such as cAdvisor may have the functionality you are looking for.


How can integration tests for micro services be written? Writing unit and integration tests for individual micro services is part of standard development processes. However, how can these services be tested with one another? And then integrated into a CI system? As well as respecting the correct ports, shared network and secrets. Should the individual micro services be versioned in the same Git repo? For context, I use docker compose with a Flask app that acts as the oauth2 server for a Grafana instance. Other instances include PostgreSQL, Redis, nginx and InfluxDB.

A simple approach would be to test the individual APIs, but what if they change on one side but not updated on the other? How can it be made sure that the APIs are being used correctly in the first place?

Well, the name "integration" test means to test different pieces of the system together, I would not call a test that only uses one component an "integration" test.

You'd wire them up like in production (in a testing environment, of course), provide test data and let them do their thing. The components themselves do not need to know - they should not know - that they are under test.

The "wiring up" should be as similar to the production environment as possible, re-using configuration code where possible, to test those bits&pieces as well. Don't put them in the same repo just because you do integration tests.

To see whether the components work together correctly, you would have a test driver that exercises several of them together. It will set up the test data and look at the end results. If everything is green, then the APIs have, by definition, been used correctly. This means that there should be plenty of integration tests - if in doubt, I'd err on the side of having too many integration tests rather than too many unit tests. Obviously this depends a lot on what you are actually doing...


Should I use release branch to push my changes? Should I use a release branch to push changes out to production or release it off master branch - the main pipeline where changes are merged by developers of their feature branches?

Not to mention, there is still some amount of manual testing and that happens on the master branch after developer has completed the changes in feature branch and automation testing, code review is completed and changes are merged to master.

What is the recommended practice?

What is the recommended practice?

The book ”Accelerate” by Forsgen, Humble and Kim documents their research that teams that use fewer long-lived branches are more successful.

The question is then what is the best minimum set of branches for a given team. The answer to that, like many things, is ”it depends”. A team that pushes into production a dozen times a day will prefer tag then release from master. A team that releases once a sprint into an integration test environment for a week before it goes live will generally prefer two branches.

  Should I use a release branch to push changes out to production or release it off master branch

My personal experience is that adding branches later if you need them is easy but removing branches is very hard. So given that we want to minimise the number of branches I recommend that small teams who release often always release from a tag applied to master. It's very easy to create a branch from the tag that was last put live if you do need to create a short-live patch release branch. The point is that you want that to be the exception not the rule give the research in the book named above.

I have seen many teams use a two branch approach. To common ones are:

Integrate code into a ”develop” branch and deploy to an integration test environment. Then cherry-pick signed-off work down to master where it is tagged to release.
Integrate code into ”master” and deploy into a test environment. Then cherry-pick signed-off onto a long-lived ”release”  branch where it is tagged to released.

These approaches are actually identical. The reason to take this “two branches” approach is so that features can be released independently and reordered.

  Not to mention, there is still some amount of manual testing and that happens on the master branch after developer has completed the changes in feature branch and automation testing, code review is completed and changes are merged to master.

There isn’t really enough information in your question about how your team actually works to know whether a one or two branch strategy would be a good fit. Your actual release, testing and support ways-of-working dictate the best strategy for your team.


copy contents of file on ansible server to another file on remote servers I have a bash script on about 70 severs in /opt/zservice/etc/config that i want to change its content with ansible. the new content is on ansible server in this path /data/new_config.
how i can use ansible (how to write a task) that reads this file new_config and then replace config content ?

something like this in bash: /data/new_config > /opt/zservice/etc/config

Copy module should work

- hosts: all
  tasks:
    - copy:
        src: /data/new_config
        dest: /opt/zservice/etc/config
        owner: root
        group: wheel
        mode: '0750'
        backup: yes

If you worry about the number of the hosts take a look at serial.


Google CloudPlatform remove/delete app deployed during tutorial steps I created a Project at Google CloudPlatform, and followed it's on screen tutorial to deploy an PHP app.

Now that it's done, how can I delete/remove the deployed app to prevent charges.

I saw an option at: "Google Cloud App Engine -> Settings -> Disable App" but does it prevent charges?

Yes, disabling an application prevents charges, but only from the App Engine prospective, there could still be other changes from other Cloud products usage. From Disabling an application:

  Your app's resources in both the standard and flexible environment are
  disabled and therefore, won't incur charges. However, billing charges
  can still occur for the other services in your Cloud Platform project,
  for example you can still incur storage charges for your stored data.

To completely end all Cloud billing for a project you can shut it down (i.e. mark it for deletion!). But please note that you won't be able to re-use its ID, ever.


How to mark a check as warning or error if a check fails multiple times within a certain time range using Sensu? A certain check works, but it should be checked multiple times within a certain range before a user gets notified.

There are multiple Sensu attributes that could be configured. This list has been consulted, but no solution was found. Is it possible to mark a check as warning or error if a check fails multiple times in a certain time range?

Perhaps aggregates would be an option, but should this not be used for multiple clients instead of using it for one check?

{
  "clients": 15,
  "checks": 2,
  "results": {
    "ok": 18,
    "warning": 0,
    "critical": 1,
    "unknown": 0,
    "total": 19,
    "stale": 0
  }
}

You want to use the occurrences filter, which is included by default in Sensu.

First, apply the filter to your handlers:

{
  "handlers": {
    "email": {
      "...": "...",
      "filters": ["occurrences"]
    }
  }
}

Then, in your check, you can use the occurrences attribute to only trigger the handler once the check fails or warns a certain number of times in a row:

{
  "checks": {
    "check-http": {
      "...": "...",
      "occurrences": 2
    }
  }
}


Using Artifactory plugin in Jenkins declarative pipeline I am using Jenkins Declarative Pipeline to automate my build process. We want to publish our artifacts to a remote JFrog repository only if certain conditions (Sonar,Checkmarx) pass.

After a bit of research, I found that Artifactory plugin is useful for this. But I am unable to find any document on how to integrate in declarative pipeline. Below is the code snippet from Jenkinsfile

stages{

    stage('Pre-Build'){
        steps{

             script{
                def server = Artifactory.server 'LocalJfrog'
                def rtGradle = Artifactory.newGradleBuild()
                rtGradle.resolver server: server, repo: 'gradle-dev-local'
                rtGradle.deployer server: server, repo: 'gradle-release-local'
                rtGradle.useWrapper = true
            }

        }
    }
}

The conditional publish is not possible with above code as I cannot reuse the server variable even if I disable auto publish.

You can have conditionals in your declarative pipeline by using the when-block inside a stage. There is a plugin called "environment injector" which lets you set variables outside of the pipeline-script which is nice. Also if you put the step below the other steps, it won't execute if they fail.

 when {
    environment name: 'pushArtifact', value: 'true'
  }
  steps{
     //push artifact
  }


Is it possible to build Jenkins jobs without using build numbers? I have a job which is running very frequently, so I'd like to avoid high number of builds. Is there any way to drop job build numbers, so the job is constantly ran, but without high numbers of builds?

It's not possible to avoid setting the BUILD_NUMBER environment variable, but you can avoid using it. Try using the Jenkins plugin Build Name Setter Plugin. It adds an option in your build configuration to set the build name to whatever you want, and you can ignore the BUILD_NUMBER environment variable this way.

Consider 1.250.5 versus 1.250.${BUILD_NUMBER}. The former will be constant no matter how many times you build and the latter will increment by one every time you build. It's also possible to use other macro values in between ${} to make more interesting dynamic build numbers.


How to make custom SLES (SuSe Linux Enterprise) Dockerfiles? To use a dockerized SLES, you need to use an official base image and run it on a SLES host.

But this does not seem to work out of the box - what needs to be done to build a custom image?

Dockerfile:

FROM suse/sles12:latest
RUN zypper ref -s
RUN zypper --gpg-auto-import-keys ref -s

Docker build output - image will be created but obviously it is not possible to install any packages because the central repo seems to be not preconfigured.

Vendor documentation: https://www.suse.com/documentation/sles-12/book_sles_docker/data/customizing_pre-build_images.html

Sending build context to Docker daemon  3.584kB
Step 1/3 : FROM suse/sles12:latest
 ---> d0f7e807658b
Step 2/3 : RUN zypper ref -s
 ---> Running in 235316f5bbe1
Refreshing service 'container-suseconnect'.
Problem retrieving the repository index file for service 'container-suseconnect':
[container-suseconnect|file:/usr/lib/zypp/plugins/services/container-suseconnect]
Warning: Skipping service 'container-suseconnect' because of the above error.
All services have been refreshed.
Warning: There are no enabled repositories defined.
Use 'zypper addrepo' or 'zypper modifyrepo' commands to add or enable repositories.
 ---> 21e8beb32227
Removing intermediate container 235316f5bbe1
Step 3/3 : RUN zypper --gpg-auto-import-keys ref -s
 ---> Running in 72ba1ad22da8
Refreshing service 'container-suseconnect'.
Warning: Skipping service 'container-suseconnect' because of the above error.
All services have been refreshed.
Warning: There are no enabled repositories defined.
Use 'zypper addrepo' or 'zypper modifyrepo' commands to add or enable repositories.
Problem retrieving the repository index file for service 'container-suseconnect':
[container-suseconnect|file:/usr/lib/zypp/plugins/services/container-suseconnect]
 ---> 6426e55c2d87
Removing intermediate container 72ba1ad22da8
Successfully built 6426e55c2d87

UPD. While this one is working:

 docker run -it suse/sles12 bash -c "zypper -n ref -s"
 Adding repository 'SLES12-Debuginfo-Pool for sle-12-x86_64' ...[done]
 .....

UPD. Finally, the following worked for me: run a container, install everything there and finally commit the container to a new image (docker commit container-id image).

As @Tensibai already indicated in one of the comments, some packages are missing:

Use 'zypper addrepo' or 'zypper modifyrepo' commands to add or enable repositories.

Either use zypper addrepo or zypper modifyrepo in your dockerfile to solve the issue.


How do I configure an AWS network load balancer with an A record in Cloudflare I am trying to use an AWS Network Load Balancer to point at a group of servers. I would then like to take my DNS and point it at the load balancer. However, after I set everything up I cannot find and ip address for the AWS load balancer. I only see a URL value but the A-record in Cloudflare requires an actual IP.

How do I configure the DNS to point at the Load Balancer?

You create a CNAME record referencing the assigned hostname of the balancer as the target.

CNAME records are not valid at the apex of a zone (i.e. the "naked" domain, without www or any other prefix) but Cloudflare allows you to create one there -- sort of -- they actually allow you to believe that you're creating one there, and then their service automatically transforms it into one or more valid A-records, dynamically.

This feature is enabled by default at the apex.

https://blog.cloudflare.com/introducing-cname-flattening-rfc-compliant-cnames-at-a-domains-root/


Is it possible to template JIRA issues with pre-filled field contents? In first place I look for default JIRA functionality e.g. without 3rd party plugins to simlify and improve the issue workflow.

Examples:

Simple: Template for an acceptable bug report
Probably, less simple: Set specific keywords if issue is created by a specific user e.g. "ops" for issues created by myself because I am currently in an Ops team. :-)

I don't think you will be able to get far without any plugins. What you can take a look at is the Default Values for 'Create Issue' screen plugin. We were using it in the past, however gave up because defaults could be set only per issue type, and no other conditions. I was in contact with the supplier, though, and they confirmed that defaults per user and other extensions were on their roadmap. Makes sense to check if they are already available.


yaml files in jenkins pipeline Just started out working on jenkins, among other things I'm trying to understand the role of yaml in pipelines. I understand that pipelines or declarative pipeline (do other type of pipelines exist in jenkins?) are defined using a syntax that is based on an extension of groovy and that yaml can be used along with some plugins or extended library to generate an on the fly groovy-based pipeline definition. Is my understanding correct?

The blog post you linked to in your comment which describes using YAML to generate Pipelines is about an unofficial 3rd-party Jenkins plugin which adds this feature; neither native Jenkins nor official plugins support using YAML to generate Pipelines.

pipelines or declarative pipeline (do other type of pipelines exist in jenkins?) are defined using a syntax that is based on an extension of groovy

There are two types of natively-supported Pipelines: Scripted Pipelines (which are pure Groovy scripts) and Declarative Pipelines (which use a Groovy-based DSL).


Can a cross platform app be run as a container on both Linux and windows hosts? I have read (and seen several online videos) that Linux containers can’t be run on windows and visa versa. It is my understanding that this is due to the fact that the container doesn’t contain a kernel and therefore it relies on the host kernel.

But if I have a simple app like a python hello world, what prevents this from running on both operating systems?

But if I have a simple app like a python hello world, what prevents this from running on both operating systems?

It won't work in the same way that copying /usr/bin/python from a Linux machine to a Windows machine and using it to run your application doesn't work.

Docker isn't running the hello world script, it's running Python, which needs to be compiled for specific operating systems and architectures.


Powershell script in jenkins is hanging I'm building a process to automate deploys using Jenkins and I have an issue getting the code on the server

I have a powershell script on the deploy server that all it does is do an hg update command.

$ciPwd = ConvertTo-SecureString $env:CI_SERVER_PWD -AsPlainText -Force
$ciCredential = New-Object System.Management.Automation.PSCredential ($env:CI_SERVER_USR, $ciPwd)
Invoke-Command -ComputerName web1-dev,web2-dev -Credential $ciCredential -ScriptBlock { C:\hg\Update.ps1 } -InDisconnectedSession -SessionName srv1,srv2

Write-Host "web1-dev output:"
Receive-PSSession -Name srv1
Remove-PSSession -Name srv1

Write-Host "web2-dev output:"
Receive-PSSession -Name srv2
Remove-PSSession -Name srv2

the script always hangs at the 'web1-dev output:' portion.

I'm kinda new to powershell scripting so its possible I'm doing something wrong, and if there's a better way to call a remote powershell script, I'm open to that

I'm guessing that your PowerShell script is hanging because of the Invoke-Command command. My best guess is PowerShell is either prompting you for something, you are passing in an argument incorrectly, or you are executing the script in your script block incorrectly. I'm not sure exactly how Script-Block works when referencing paths to PowerShell scripts, but it could be because C:\hg\Update.ps1 does not exist on the remote server. It could also be because of the commands in your "C:\hg\Update.ps1" script. Some debugging tips that I recommend:

Remove your Invoke-Command line and see if the program executes.
Replace the ScriptBlock command with a simpler command, like Get-Culture, command and see if it still hangs.
Replace your C:\hg\Update.ps1 with another script that performs a simple command.
If all of those options still hang, try using the simple script method, but with different args.

And of course as with all PowerShell remoting issues, you should check that your firewall and network settings on the remote server allow connections from Jenkins over ports 5985 and 5986.


How to deploy SQL Script in DevOps culture and Rollback situation too? How we can manage database part in DevOps environment/CICD pipeline, Here taking about the SQL Script. For Example any deployment where Insert and Update SQL's, If we setup in CICD pipeline then I will execute everytime thus I will re-execute everytime & DevOps scenarios are fail in this situation.

Overall How we can setup pipeline with SQL script and Application in real world and In case of rollback, How can we manage it ?

There are a few different things you can do:

In your CI process, run your new SQL update scripts against a Dockerized SQL Server. This will allow your CI server to reject the changes if there are basic errors in the script.
If you are fully containerized look into tools like Kubernetes.  These tools have ways to handle rolling back failed production updates.
Utilize a Blue Green Deployment strategy along with integration testing.  This provides a fallback in case things go wrong and allows for different types of automated testing schemes in your CI process.


Creating a cron job with Ansible (ERROR! 'cron' is not a valid attribute for a Play) I am relatively inexperienced when it comes to Ansible however I have previously created playbooks which will create a cron job successfully. For the life of me I cannot work out what is wrong with my .yml file.

I am receiving the below error when attempting to run my playbook locally with:

ansible-playbook -i "localhost," -c local crond.yml

ERROR! 'cron' is not a valid attribute for a Play

The error appears to have been in
'/etc/ansible/cds/roles/backups/tasks/crond.yml': line 3, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- cron:
  ^ here

I have seen similar questions asked on SE/Ansible forums but to no avail. Any help/guidance would be much appreciated.

Here is my crond.yml file:

- cron:
    user: "root"
    minute: "30"
    hour: "02"
    job: '{{/usr/bin/rsync -avhz -e "ssh" --rsync-path="sudo rsync" -H --delete --numeric-ids <ipaddress>:"/data/rsnapshot-backups/<ipaddress>" /data/rsnapshot-backup}}'
    state: present

crond.yml in your example is not a playbook. It is a tasks file (list of tasks), while playbook is a list of plays.

A play must have hosts directive at minimum. Please see http://docs.ansible.com/ansible/latest/playbooks_intro.html#hosts-and-users

This is a playbook:

- hosts: localhost
  tasks:
    - cron:
        user: "root"
        minute: "30"
        hour: "02"
        job: '/usr/bin/rsync -avhz -e "ssh" --rsync-path="sudo rsync" -H --delete --numeric-ids <ipaddress>:"/data/rsnapshot-backups/<ipaddress>" /data/rsnapshot-backup'
        state: present


node js deployment on production server I wanted to know standard process to deploy nodejs app on production. As of now I am following the process as below

Pull the source code

git pull git_url

Install npm dependencies

npm install

Build the code

npm run build

run the application

nohup node server/index.js &

But I am thinking of pulling source code to prod is the totally wrong way. Please, someone, guide me the exact way to deploy node app on the production server. Also, any other way to run the app continuously as service. I am aware with pm2 module to run a process in the backend. If there is any other way to manage it properly.

You could issue an npm pack which will create a tgz and then you could push it to an artifact manager like Nexus3. By doing this one uses the same approach as creating artifacts in Java like JARs and WARs. If you ensure that the artifacts cannot be overwritten, you specify a version and restrict publication to an artifact manager by the CI like jenkins then the artifacts are immutable and you are in control. E.g. if there is a bug in version X you could revert it to the previous version that was working well.

Note: if you are planning to use npm pack I would suggest to use an abbreviation like @angular does to ensure that there is a clear destinction between artifacts that were created by your company and libraries from npmjs.org. This could be done by adding an abbrevation of your company like @z to the package name. A name could look like: @z/some-package.


Rename build pipeline .yml files in Azure DevOps I have a .NET Core application that has three build pipelines. (Windows, Linux, macOS)
In each of them, the application gets assembled a little different. This part works so far.

Azure created for each pipeline a .yml file in the root folder of the repository.

azure-pipelines.yml (Linux Build)
azure-pipelines-1.yml (Windows Build)
azure-pipelines-2.yml (macOS Build)

I would like to rename them to something like azure-pipelines-platform.yml

But when I do so, the configuration in Azure gets broken. I can rename the build configuration, but not the file. And I find no way to configure the link between .yml file and Build config in DevOps

Is it generally not possible to rename the .yml files?

As suggested by @030, I have prepared two screenshots that show how to connect the .yml files with the build pipeline.

Rename the .yml file in the repository
Open the pipeline editor

Select the file from the dropdown

Select the in step 1 renamed *.yml file and click on save.

After the existing .yml is assigned, the pipeline works as before the renaming.


Terraform foreach multiple map variables I need to do the following in Terraform but can't work out how
Here is what I have
local {
app_config = {
    test-web = {
      name            = "test-web-${local.environment}"
      websockets      = false
      subnet          = "backend"
      audience_subnet = "frontend"
      url             = "https://test-web-${local.environment}.mydomain.com"
    }
    test-api  = {
      name            = "test-api-${local.environment}"
      websockets      = false
      subnet          = "backend"
      audience_subnet = "frontend"
      url             = "https://test-api-${local.environment}.mydomain.com"
    }
  }

keyvault_secrets = {
    aat = {
        test-api = {
            application_id = "1111"
            client_id = "2222"
            administrator_pass = "3333"
        }
        test-web = {
            application_id = "4444"
            client_id = "5555"
            administrator_pass = "6666"
        }
    }
    Demo = {
        test-api = {
            application_id = "1212"
            client_id = "2323"
            administrator_pass = "3434"
        }
        test-web = {
            application_id = "4545"
            client_id = "5656"
            administrator_pass = "6767"
        }
    }
    Dev = {
        test-api = {
            application_id = "9999"
            client_id = "8888"
            administrator_pass = "7777"
        }
        test-web = {
            application_id = "9898"
            client_id = "8787"
            administrator_pass = "7676"
        }
    }
}

resource "azurerm_key_vault_secret" "app_id" {
  for_each = var.apps_config

  name         = var.apps_config
  value        = each.value.application_id
  key_vault_id = data.azurerm_key_vault.mykv.id
}

resource "azurerm_key_vault_secret" "client_id" {
  for_each = var.apps_config

  name         = var.apps_config
  value        = each.value.client_id
  key_vault_id = data.azurerm_key_vault.mykv.id
}

resource "azurerm_key_vault_secret" "admin_pass" {
  for_each = var.apps_config

  name         = var.apps_config
  value        = each.value.administrator_pass
  key_vault_id = data.azurerm_key_vault.mykv.id
}

What I can't work out is how you do the following:
For every app_config defined, for example:
test-web
test-api
A Key Vault secret are created by the environment defined when running:
Terraform plan or apply -var="environment=dev"
I want it to create the 3 secrets for each app_config

Is something like the below what you are looking for?
locals {
  apps = keys(var.app_config)
}

resource "azurerm_key_vault_secret" "app_id" {
  for_each = toset(local.apps)

  name         = each.key
  value        = var.keyvault_secrets[var.environment][each.key]["application_id"]
  key_vault_id = data.azurerm_key_vault.mykv.id
}


How can Jenkins help to implement CI/CD for Drupal? Assume you want to implement continuous-integration (= CI) and continuous-deployment (= CD) for building and maintaining websites using Drupal (let's assume for Drupal Version 7). Often times Jenkins seems to be recommended as one of the most appropriate solutions for doing so.

But what kind of CI/CD related functionality can I actually use it for? Here are some of things I'm thinking of, but I'm not sure if I'm correct on that, and/or if these are the only things:

CI related: Automated testing.
CD related: Migrating custom module development between dev / staging / QA environments.

What you've listed is actually two different things, continuous delivery and continuous integration.  When people talk about CI they typically are only referring to the testing part; automated unit, function, and integration testing with some sort of version bump or code promotion at the end.  Looks like the Jenkins wiki has some specific pointers about Drupal.

As far as environment promotion, you're getting into continuous delivery which Jenkins can also be used for.  Consider looking at the more modern Jenkins Pipelines for CD pipelines.


Add permissions to readers (DevOps Web Portal) I have a public Azure DevOps project, and I want to allow everyone to add Work Items.

However, adding the Readers group as a member of Contributors for the permissions doesn't change anything, and I can't find "Add Work Items" but only the "Delete Work Items" toggle for changing its own Readers perms so I am really stuck here.

The contributors perms:

Having to add every mail account to allow people to add work items isn't possible in my situation (Game Engine project) so I really have to find a solution.

Usually each projects would have teams defined with the team members. So if you have a team defined then add the team to the contributors list.


Does `npm audit` add any value when using `dependabot` Context
There are multiple ways to scan projects for vulnerabilities.
Dependabot can be configured to check repositories for issues, and automatically submits pull requests to resolve.
NPM Audit will scan the packages used in an NPM solution for known vulnerabilities.
We're trying to work out whether, if Dependabot is enabled, there's any added value to using NPM Audit in our pipelines.  I'm asking this solely from the perspective of what's detected; not how the tools work (i.e. whether they can cause a pipeline to block/fail).
The actual question
Do both tools base their decisions on some common known-issue database, or is it common to see each tool detect different sets of problems?

Dependabot and npm audit both poll the Node Security Working Group database for Node-based projects. However, Dependabot has the added ability to check dependencies in numerous other types of projects as well.
Also, each report Dependabot generates includes useful info and links directly to a GitHub Advisory Database listing (e.g., CVE-2017-16021) that itself has multiple links to other listings such as NIST. On the other hand, npm audit's reporting is limited to a single link to an npm advisory listing with far less info (e.g., #100).
Overall, Dependabot covers all the bases — and more — that npm audit does. No need to use both unless you need a customized audit flow in your package scripts.


How to deploy static website from public repo via Github Actions to private web server I like to start an open-source project via Github where topics (markdown pages) can be created from the community. Via PR the changes get merged to the main repo and then it triggers a Github Action workflow. Inside this workflow, the markdown input gets processed to static HTML pages which should later get deployed to an Nginx web server.
The current (prototyped) deployment is done via SCP for GitHub Actions and SSH for GitHub Actions. For both actions, I have to provide the user and password inside the workflow.yml file.
To make the project public I need another approach, where others cannot get the credentials.
How can I realize my idea?

By coincidence, I have found a possible solution via Github Secrets.


DevOps tools and automation of manual processes This is a general question, but couldn't find a better place to ask. I'm a new DevOps member in a company, and I've been ask to "update" our DevOps technologies.

Generally speaking, in what areas do we need to provide tools, and what are some latest and advanced technologies/tools to do that?

For instance, we need container management (e.g. docker, kubernetes), log analysis (e.g. ELK Stack), Application monitoring (e.g. JMX), Weblogic, database management, Jenkins, etc.?

What else, and what tools? Which processes are probably needed to be automated? It can be anything. I just need some advice.

Alright... you're not going to implement every tool or automation at the same time, as some of them have quite deep impact on your development processes (and I daresay, development culture). Take a step-by-step approach; research each individual tool. Figure out what it actually does, what it is useful for. Play around with them, install them locally, go through tutorials, and see what they do, or whip up a quick VM somewhere (AWS, etc.) if you need more than one.

Then, take a look at the work your team/company is doing right now. What are the manual process you wish to automate? Ideas should spring up pretty quickly.

A nice, simple(-ish), bog-standard stack to start out with:

Vagrant to automate setting up VMs for your developers, or test environments.
Ansible to automate configuration management (relatively lightweight compared with Chef or Puppet)
Gitlab or Jenkins as your automated CI/CD driver, mostly depending on taste
Docker for your containers
docker-compose, docker-swarm, Kubernetes, OpenShift to automate container deployment
ELK or Splunk for your logging/monitoring, again depending on taste

Have fun!


simulate SQL DB usage pattern of a typical cloud native web app with sysbench? As far as I understand sysbench is for bencharking / test max. performance of a database server. I wish to simulate DB usage pattern of a typical cloud native web app (Twelve-Factor App). Is this possible with sysbench? If yes how? Or any other Open Source tool you recommend?

I work in DBaaS (MariaDB) so can't say what exact app (shop, historic info, social media, forum). I wish to deploy app on Cloud Foundry.

... [I] can't say what exact app (shop, historic info, social media, forum)

This makes this question very difficult to answer. This is because you have two basic types of queries:

OLTP and OLAP queries

  Online transaction processing (OLTP) is information systems that facilitate and manage transaction-oriented applications, typically for data entry and retrieval transaction processing.1

These kinds of queries tend to be very high volume, but short, and targeted and are a great fit for a system like MariaDB. On the other hand,

  Online analytical processing, or OLAP, is an approach to answering multi-dimensional analytical (MDA) queries swiftly in computing. OLAP is part of the broader category of business intelligence, which also encompasses relational database, report writing and data mining. Typical applications of OLAP include business reporting for sales, marketing, management reporting, business process management (BPM), budgeting and forecasting, financial reporting and similar areas, with new applications coming up, such as agriculture.2

These types of queries perform extremely poorly at scale in Maria DB unless it is properly tuned and certain database management and retrieval techniques are used (such as partitioning, for example). These performance issues are largely a result of a confluence of disk I/O issues and the fact that queries are single-threaded in terms of CPU. While sysbench can do a great job of testing the performance of OLTP queries, for OLAP queries, you will need something else. For example, one programmer forked sysbench to add the ability to do OLAP queries.

Conclusion

Unless we can get a bit more information about how much data you are working with and what kind of I/O operations you will be doing (reporting or transactional) it is hard to say if this specific tool will do the job. Ultimately database management is a complex discipline and there is a reason DBAs are able to make careers out of just database management. I have seen a lot of programmers think that they don't really need to worry about database management and they don't really need to integrate a DBA into their team. As their app scaled, they very quickly found out they were dead wrong. If you are planning a multi-tenancy DBaaS environment, you are going to need to architect your system very carefully in order to prevent one tenant from affecting the performance of the other and consider offering DBA consulting if your customers begin to suffer performance issues - and factor that into the price. If they can't afford their own databases and find that managing that in house is untenable and unaffordable, you will need to provide both the platform and the expertise as part of your service as one customers performance issue can easily become all of your customers' performance issue. That means you are going to really need to know your stuff or make sure you hire people who do. In short, databases are hard.

1 https://en.wikipedia.org/wiki/Online_transaction_processing

2 https://en.wikipedia.org/wiki/Online_analytical_processing


Azure - Connection String in Key Vault vs Application Settings For Azure Functions and WebJobs, is there any benefit to putting connection strings as Secrets in Key Vault instead of putting them directly in Application Settings (and referencing them using ConfigurationManager.ConnectionStrings)? Are Azure Key Vault Secrets primarily meant for VMs and such rather than Azure Functions and WebJobs?

It seems like it's just adding an extra step in both development (update the Secret in Key Vault and the Secret Identifier in Application Settings) and an extra step in runtime (an additional retrieval from Key Vault), with the only benefit being that the Secret in the Application Settings is an identifier instead of the actual Secret. I don't see the security benefit here, whereas there are detriments.

The benefits as I see them are the general reasons to use Azure Key Vault

Secrets are centrally stored with options for authorization, auditing, etc.
Azure can be scripted to update the secrets on a timer - so in case a connection string gets in to the wrong hands, it is only valid for a day (or a week, or whatever)
In case the secret is shared between multiple applications, you only have to update it in one place (not the greatest benefit, but still nice)


Why not run a stateless Hashicorp Vault in production? I'm contemplating whether to provision Hashicorp Vault from code, from scratch, and seed it with keys to be shared between a few, predefined consumers (some humans, some services). The provisioning step has secure access to keys already and can use them to initialize Vault. My reasoning is that this would make it operationally simpler to run -- no backups or risk of configuration drift -- and can act as an on-ramp to using Vault in more places.

Is this feasible? In Hashicorp's documentation for Vault, the in-memory storage option is not recommended for production use because the data will be lost on shutdown. Is there data that is lost that can't be re-provisioned every time?

I think you're misunderstanding what is considered bad practice.

Running vault with an in memory backend and being able to spin it up and provision it is fine. What is Not Fine is seeding the secrets in Vault from an insecure source.

If you are just using Vault as a way to simplify operations, as opposed to a long term persistent state store - then go for it.

One caveat - if you use the capabilities in vault to rotate keys for other integrated services then if you lose your vault cluster then you have just lost the keys to those services, so hopefully anything accessed using a Vault managed, regularly rotated, key is also easy for you to reprovision


Load balancing kubernetes cluster on budget KVM provider? There are a lot of cheap KVM providers these days that offer simple nodes with just a single external IP per node, without any load balancing.

What is a typical budget load balancing solution for this scenario?

I am planning to use either docker swarm or kubernetes as a container orchestration solution.

It depends on how much time you can invest into configuration and your configuration.

I have built my own cluster on AWS using Nomad and Consul where Fabio was used as Application Load Balancing layer (AWS ALB was lacking for my purposes).

Alternatively if you use k8s and do not need additional ALB you can use DNS RR (multiple A or AAAA records for your domain) or create HAProxy cluster in front of your application.