How do I make NFS volumes reliable in Docker? I host various docker containers on my Ubuntu 18 machine. A few of them require storing their data on my Synology NAS. At first, I was using the host machine's /etc/fstab to control NFS mounts, which I then mounted in the containers (as a mount, not a volume).

However, I figured it would be a better idea to have the containers map NFS to mount points in the containers. The host really has no use for the mounts, so it didn't make sense to maintain them there.

At the moment, I am configuring my NFS volumes like this (using Docker Compose v3 format):

volumes:
  data:
    driver_opts:
      type: nfs
      o: addr=192.168.1.51,nolock,soft,rw
      device: :/volume2/nextcloud

This works great when the NAS is booted and working normally. However, I had a power outage and noticed all sorts of problems. Also, the timing of which machines boot first (NAS vs Ubuntu box) affects reliability of my docker container volumes. In my last situation, the NAS was not powered on. So when the container was started, it failed:

  ERROR: for app  Cannot start service app: error while mounting volume '/var/lib/docker/volumes/nextcloud_data/_data': error while mounting volume with options: type='nfs' device=':/volume2/nextcloud' o='addr=192.168.1.51,nolock,soft,rw': no route to host

What would be nice is if docker would keep retrying to mount the volume until the NAS was powered on again. That would make this hands-free and prevent any timing issues (on which devices boot first across the network) from causing permanent failures like this.

I'm also not sure what happens if a volume is created, and the NAS is powered off at any point. Does the volume stay available? Does docker keep trying to reconnect the NFS mount? I feel like there is very little control here.

Note I just use Docker Compose. I don't use Swarm for technical reasons I won't go into here. Can someone recommend a way to resolve these reliability issues? Are NFS volumes in Docker the way to go? Should I go back to mounting on the host? Is there an amazing 3rd option out there somewhere?

There are options in docker that will retry the startup of a container, restart in version 2 and restart-policy in version 3 (you'd need compatibility mode enabled for the version 3 syntax to work). However, I believe they only work when the issue is from the application inside the container fails, not when there is an issue creating the container like you see with a volume mount failing (or would also happen if the image couldn't be retrieved from a registry).

To handle the failing volume mount, I believe swarm mode is your best option despite your objections. You can create a single node cluster with docker swarm init and deploy your compose file with docker stack deploy -c docker-compose.yml stack_name, making an easy transition from docker-compose. Swarm mode looks at the overall state of the service and continuously tries to make the current state match your target state (as defined in the compose file), which will handle a failing volume mount that eventually corrects itself. I don't have an NFS server to test on right now, but here's a scenario with a bind mount to a missing folder:

$ cat docker-compose.vol-bind.yml
version: '3'
volumes:
  bind-test:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /home/bmitch/data/docker/test/missing
services:
  bind-test:
    image: busybox
    command: tail -f /dev/null
    volumes:
      - bind-test:/bind-test

$ docker stack deploy -c docker-compose.vol-bind.yml voltest
Creating network voltest_default
Creating service voltest_bind-test

$ docker service ls
ID                  NAME                   MODE                REPLICAS            IMAGE                                             PORTS
omzaeo7mrour        voltest_bind-test      replicated          0/1                 busybox:latest

$ docker service ps omzaeo7mrour
ID                  NAME                      IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR                              POR
TS
kpz0l79eucaw        voltest_bind-test.1       busybox:latest      bmitch-asusr556l    Ready               Ready 2 seconds ago
j6fylzhvcv60         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 5 seconds ago    "starting container failed: er…"
61o6raohp0xl         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 12 seconds ago   "starting container failed: er…"

$ docker inspect kpz0l79eucaw
[
    {
        "ID": "kpz0l79eucaw1856obmwvcak1",
        "Version": {
            "Index": 445
        },
        "CreatedAt": "2019-04-29T12:57:25.925788528Z",
        "UpdatedAt": "2019-04-29T12:57:34.3467203Z",
        "Labels": {},
        "Spec": {
            "ContainerSpec": {
                "Image": "busybox:latest",
                "Labels": {
                    "com.docker.stack.namespace": "voltest"
                },
...
        "Status": {
            "Timestamp": "2019-04-29T12:57:33.936048295Z",
            "State": "failed",
            "Message": "starting",
            "Err": "starting container failed: error while mounting volume '/home/var-docker/volumes/voltest_bind-test/_data': failed to mount local volume: mount /home
/bmitch/data/docker/test/missing:/home/var-docker/volumes/voltest_bind-test/_data, flags: 0x1000: no such file or directory",
            "ContainerStatus": {
                "ContainerID": "4c75c851bd43b5ef57b4785a4611f78501279933822eab90c693e1108f53ee82",
                "PID": 0,
                "ExitCode": 128
            },
            "PortStatus": {}
        },
...

$ mkdir missing

$ docker service ps omzaeo7mrour
ID                  NAME                      IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR                              POR
TS
okylli7smu1c        voltest_bind-test.1       busybox:latest      bmitch-asusr556l    Running             Running 4 seconds ago
9tpm188ysu2k         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 14 seconds ago   "starting container failed: er…"
kpz0l79eucaw         \_ voltest_bind-test.1   busybox:latest      bmitch-asusr556l    Shutdown            Failed 21 seconds ago   "starting container failed: er…"

From the above, you can see as soon as the missing directory was created, the bind mount succeeded, and the container that failed to be created as retried and started successfully.


How to check AWS spot pricing automatically? I've been thinking about a Jenkins job to check the pricing for the agent boxes we spin up; every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc.

My first instinct was that the best route is to have an hourly Jenkins job that runs a aws ec2 describe-spot-instance-requests and checks for failed requests (and then slacks us on a failure). But I'm wondering if there's a cleaner method that involves actually comparing hard prices (and could therefore tell us exactly what's off, and by how much), rather than looking at successful/failed requests.

Anyone set up something similar? How'd you do it?

Spotted an open source tool called autospotting that just might help with this:

  Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand instance is detached from the group and terminated.

We have this in our to-do pipeline, will be able to add more context once we finish that.

Update:

Another tool that was recently demo'ed in a conference was mapbox/spotswap

This works slightly differently. It monitors a normal ASG with either On-Demand or Reserved instances and then, if scale arises, bids and provisions spot instances of similar compute level on a separate ASG.


Where should I store my build configuration files? At my work we use git for our source control, MSBuild for our builds, and Jenkins for our CI. Our git repo will have different branches for the cycle of the product (develop, QA, release).

My question is where should I store the msbuild xml? If I check it in alongside the product, say in develop and there's a feature branch, the file gets merged when a team does a merge from the parent. So that's not an option.

I also want the build config to stay with the version of the product. So the build.xml will go from develop->QA->release.

How is everyone else handling this?

Normally the build configuration needs to stay consistent with the product code, so it should be stored into the same git repo as the product code.

It's true, the file would get merged when a branch merge from the parent is performed.

But often it's a trivial (empty/fast forward) merge - if there were no changes to the msbuild xml file in the parent branch.

And if the merge is not a trivial one it means there were changes in the build process in the parent, which need to be taken into account in the child branch as well since it also picks up the code updates that go with those build changes. In other words the msbuild xml merge is required in such case to avoid build breakages.


How to deploy MySQL server repeatedly From fifty-thousand feet, how are, for example, multiple MySQL containers handled?
In the context of my limited experience, simply running the container is insufficient, with additional configuration required.  It just seems not quite so repeatable if, after running the container, it must be further tinkered with before any CRUD operations.  Or is that just the process?

I think it depends on the context.
I feel your pain with regard to needing to tinker and tweak a MySQL/MariaDB instance. It's not extremely difficult to get an instance up and running and THEN populate the DB from, say, a mysqldump by attaching volumes to the MySQL instance either using a Dockerfile for the instance or by "manually" bash-ing into it and loading your tables, tweaking your network config, setting up complex user permissions, etc.
But it's usually a STRUGGLE to deploy a "ready to go" mysql instance from a docker-compose.yml in my initial experience using Docker Composer for a MySQL container in an app deployment, but it IS doable.
What kind of scale are you trying to achieve? Do you NEED Kubernetes or Docker Swarm? I think in the case of most apps, the answer is honestly no. You can almost always come up with a crafty workaround with ba(sh) scripts or RUN/CMD (Dockerfile) or command: (docker-compose).
Are you trying to support an automated deployment? or just get an app up and running? or trying to increase resilience? Increase capacity at scale?


Chef 12: Error while uploading cookbook I am using Chef and trying to upload a cookbook on chef-server but getting an error in response:

  {"error":["Invalid key all_files in request body"]}

However, I can see the list of cookbook using knife node list

My chefDK version:

chef -v
Chef Development Kit Version: 2.0.28
chef-client version: 13.2.20
delivery version: master (17c1b0fed9be4c70f69091a6d21a4cbf0df60a23)
berks version: 6.2.0
kitchen version: 1.16.0
inspec version: 1.31.1

Chef-server version is 12.0.5

Logs:

DEBUG: Response body contains:
{"error":["Invalid key all_files in request body"]}
DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_response

I can't upload more logs, if one needs particular info then I can provide.

Any help?

This question seems to have been answered in the Chef issue. To close the loop, I will summarise it here:

This was a bug introduced in Chef 12:

  The issue is with chef_solo vs chef_zero. It seems chef_solo was the old way of doing Chef things. So I added chef_zero.enabled true in the knife.rb file to activate chef_zero instead of chef_solo. When you do this, you should be able to upload.

It was subsequently fixed in Chef 13:

  This is fixed in Chef 13.3.3, which will be released in the ChefDK 2.2 release

At the time of writing, Chef is in major version 15, so this issue is very unlikely to be encountered in the wild.


How to get server url from kub dashboard? I have access to kubernetes dashboard on aws with token. And now I want to connect with kubectl, how to get server url for kubectl config?

When running an EKS cluster, the best way to configure your ~/.kube/config is by using the SDK's update-kubeconfig:

aws eks update-kubeconfig --name your-cluster-name

Make sure to configure your credentials as AWS_* environment variables. Or even better use something like aws-vault.


Running task or role before any playbooks Is there a way to run a specific task or role at the very beginning of every ansible-playbook run?

I'm looking for something that would go into ansible.cfg or something like that, so all playbooks using that file would run a specific task at the beginning.

My intention is to use it for doing some verifications: for example, "is the local host listed on the inventory?  If not, perhaps I'm using the wrong inventory".

I could import or include such a role in every playbook, but I want it to be automatic, and to work even for playbooks that don't know about that role.

What you are looking for is the meta/main.yml in the roles of Ansible.

So make your every playbook (the ones you want to run after a task/role) as a role itself (to make it secondary execution) and what ever task/role you want to run before it should be added to the meta/main.yml of all the secondary execution roles. This would make all your roles execute the role mentioned in meta/main.yml


How can I force Docker Compose to rebuild containers when dockerfile changes? I am using docker-compose to start 2 containers using the following docker-compose.yaml.

version: '2.4'

services:
  myservice1:
    container_name: mycontainername
    build:
       context: .
       dockerfile: ./my-dockerfolder/Dockerfile
       args:
       - MY_ARG=${MY_ARG}
    environment:
       - MY_ARG=${MY_ARG}
    mem_limit: 3500M
    ports:
    - "9090:9090"
    extra_hosts:
    - "one.mydomain.net:127.0.0.1"
    - "two.mydomain.net:127.0.0.1"

  myservice2:
    container_name: myothercontainername
    build:
       context: .
       dockerfile: ./other-dockerfolder/Dockerfile
       args:
       - BUILD=${BUILD}
       - MY_ARG=${MY_ARG2}
    environment:
       - MY_ARG2=${MY_ARG2}
    ports:
    - "2023:22"

My problem is that when I run docker-compose up again, it is using the same image to create the container, regardless of changes to the docker file.

How can I force it to take changes to the docker file into account?

Are other docker objects being reused that I need to be worried about?  I want the second time I run docker-compose up to be as clean an environment as the first time.

note: I do not want to delete images that I do not need to delete.  I have a slow connection to the docker repo, and removeing all the images would make my docker compose up take ~22 minutes.

Why not docker-compose build ?

https://docs.docker.com/compose/reference/build/

You can also do docker-compose up --build to force a rebuild.


Ansible playbook to ensure Ruby gems are installed Can I use ansible to make sure certain Ruby gems are installed on a list of Windows 7 hosts? How do I do this?


How to install aws cli on a self-hosted azure DevOps Windows agent? I have quite some difficulties installing aws cli on a self-hosted Azure DevOps Windows agent as part of a pipeline. Does anyone have a working solution?

What I have tried so far:

a) install aws cli via pip

Firstly, I install Python (tried both; Python2 and Python3) within the pipeline (Powershell) by downloading and extracting a tar file. Example:

(New-Object System.Net.WebClient).DownloadFile("$(python3_download_url)", ".\Python.tar")
Expand-7Zip .\Python.tar .

Secondly, I use the freshly installed Python for installing aws cli. Example:

$(PYTHON364EXE) -m pip install awscli

Aws cli is then installed in the SCRIPTS directory, as expected.

When I try to execute aws.cmd, I receive an error message:

File association not found for extension .py
'""' is not recognized as an internal or external command,
operable program or batch file

Similar issues are described here: 1, 2, 3. The suggested solution is, to associate python files with the freshly installed python executable:

assoc .py=pyautofile
ftype pyautofile="$(PYTHON364EXE)" "%1" %*

I do not feel this is appropriate for an azure agent:

It does not feel right to use Admin permissions for a simple build
I am worried that the proposed change would impact all agents and the entire host server, while I need it for only one specific agent.

Another option would be, to install the required tools system-wide (python, aws). That would, however, make it difficult to try out a specific (new?) version of python or aws cli.

b) install aws cli via extracted msi package

... extracted the contents of the msi package via lessmsi, then run the aws cli:

C:\AWS> aws
failed to create process.

This obviously does not work either. - It actually does work properly with aws cli version 2, but not with aws cli version 1. Version 2 is not recommended for use in production yet.

c) Use AWS Tools for Microsoft Visual Studio Team Services

This is an extension for Azure DevOps, maintained by AWS. It works, but a) we have legacy scripts of a certain complexity using aws cli we cannot migrate, and b) there might be features available in aws cli which are not available in the extension.

d) call aws as python module

$(PYTHON364EXE) -m awscli --version

This works, but it means I would have to adapt and modify the legacy script.

The best way to have AWS CLI tooling installed to your custom agent is by installing from Visual Studio Marketplace. AWS CLI is part of AWS Tools for Azure DevOps Service and Azure DevOps Server (the on-premise Azure DevOps) under the name of "AWS Tools for Microsoft Visual Studio Team Services".

Installing this AWS Tools will enable AWS tooling on CI (build pipeline) and CD (release pipeline).

This is the official landing page of AWS Tools in Visual Studio Marketplace:
https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-vsts-tools

Official documentation is available at: https://docs.aws.amazon.com/vsts/index.html


Jenkins executes Selenium test in headless mode while use `Jenkins as Service` I have recently installed the Jenkins on Windows 10 Machine (Jenkins as Service) and configured one Maven Project (Selenium project which open chrome browser and executes some testcase). If i build the same, all my testcase gets execute in Chrome headless mode instead of opening actual browser.

I've tried below solution :

Checking jenkins service option to Allow service to interact with desktop

Setting NoInteractiveServices flag to 0 in Registry

But nothing seem to work.

while i use Jenkins by executing Jenkins.war file it runs the testcase as expected (non-headless) but not Jenkins as Service.

Any idea why this is happening ?

Following command line if you plan to run the same application in both a headless and a traditional environment: more details

java -Djava.awt.headless=true
You can add environment variable in Jenkins configurations
Manage Jenkins -> Configure System -> Global Properties -> Environment Variables section As:
Name :  DISPLAY
Value:   0
Installing Jenkins master as a Windows service. This will run browser in other users account to get rid from this do following things:

3.1 Go to your services and stop Jenkins service. To see all your services run
services.msc, from there find Jenkins service and disable\stop it.

3.2 Go to jenkins home directory(C:\Program Files(x86)\Jenkins) and start
cmd with administrator mode and hit
java -jar jenkins.war

if still your problem persist go through following threads
1. for as service,

2.  for general queries


Docker container host machine communication Hi i’m trying to communicate with mongodb in the host from a flask container…
So using “docker.host.internal” on the URI didn’t work (host machine is Ubuntu)
Any suggestions ?? To solve this

You can run your flask container using the host networking mode. This way docker won't isolate the container and will run it the same network namespace as the host, and you container will be able to discover your database using localhost as mongodb hostname
docker run --network host <flask-image>


Unable to Trigger Downstream Jenkins Job Based on the Promoted Builds Plugin I have a request from a user who is currently using the Promoted Builds plugin for Jenkins. This user would like Job_B to be triggered when a build from Job_A has been promoted.

In Manage Jenkins > Configure System, there are three "Promotion Levels" defined.

When I navigate to Job_B > Config, check the "Build when another project is promoted", and enter the name of Job_A in the Job Name box. However, the "Promotion" drop-down does not populate with any of the three promotion levels.

Job_A is a Freestyle job, while Job_A is a Pipeline. I thought this was the source of the issue, but when creating a test FreeStyle job, I still wasn't able to see anything populate the Promotion drop-down. Any idea what I may be missing?

Job chaining is a CI/CD anti-pattern and one of the big drivers behind developing pipelines. If the next step in the equation is to do more work, just make it a stage in the pipeline. If you need separation for whatever reason, make the freestyle job a pipeline job and add a post->success step.

https://jenkins.io/doc/book/pipeline/syntax/#post


When would I use docker-compose's "--service-ports" flag? Docker-compose's run reference includes the "--service-ports" flag.

 --service-ports       Run command with the service's ports enabled and mapped
                      to the host.

Source: https://docs.docker.com/compose/reference/run/

The description doesn't really tell me anything. Ports are mapped to the host on docker-compose up if I specify them in the docker-compose file. No matter if the flag is used or not.

What is it for?

According to the Docker documentation:

docker-compose run runs a one-time command against a service. [...]
Commands you use with run start in new containers with configuration defined by that of the service, including volumes, links, and other details. However, there are two important differences.
First, the command passed by run overrides the command defined in the service configuration. [...]
The second difference is that the docker-compose run command does not create any of the ports specified in the service configuration. This prevents port collisions with already-open ports. If you do want the service’s ports to be created and mapped to the host, specify the --service-ports flag:
docker-compose run --service-ports web python manage.py shell
Alternatively, manual port mapping can be specified with the --publish or -p options, just as when using docker run:
docker-compose run --publish 8080:80 -p 2022:22 -p 127.0.0.1:2021:21 web python manage.py shell

This is meant so that you can run singular commands against an already defined service or services using docker-compose run.
For example, lets say you set up a compose file that defines some kind of service. To start the service to run in the background you use docker-compose -f <FILE> up --detach but if you wanted to look inside the container using a terminal, you would run docker-compose run bash.
The reason for the ports being ignored is docker-compose run assumes the container is already set up, and has it's ports set up correctly. If you ran a script that did something involving ports using docker-compose run /path/to/script.sh you might not want it to change the ports that were correctly mapped with your docker-compose up command. This way, the option makes it so you have to tell docker explicitly "I actually want port mappings to change".


AWS elb vs google cloud for multiple small nodejs mini-apps deployment Currently i use AWS ELB --single instances do deploy & manage multiple versions of the same nodejs app (over 10), which differ only by the configuration. Each app is running some endless process 24/7. I want to be able to manage (start/stop/update) them separately and efficiently. Currently it's very easy to do so with elb deploy command.
However, the costs went seriously up lately (while the load increased just by 10-20%). About half of the cost is the data transfer and 40% is the EC2 instances themselves (t2.nano).
Would the use of Google Cloud be more cost efficient? and more importantly, is there a similar way do deploy and manage 10x instances of the same app? Sure when i finish reading all the docs, i might have an idea, but hoping to get some advise about the direction of research or probably about the solution itself.

you can try google cloud run. here you can run containerized applications on a fully managed serverless platform. You might have to containerize your application.
But it can help you reduce the cost as  the min container can be 0 (zero) and if any to your app is there cloud run can automatically scale to meet the demand.


Copying a file with special character & spaces I have been trying to copy a file from my remote server to AWS s3, but the file name is quite weird and I am not able to figure it out what can I do. I tried using escape characters but that did not work.
Filenames:-

# R?glement de zzzzzxxxxx v1.0 #.pdf
# R?glement de xxxxxyyyyy v1.0 #.docx

My locale is like this :-

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

I am using below command

aws s3 sync file_name s3://bucket_name/raw_files/ --sse

Please suggest

Note that the suspect filename character may not actually be ?, that could be just a filename displaying artifact if your system somehow doesn't properly display that character. BTW, R?glement sounds awfully close to the French Règlement.

One possible approach to accessing those files without worrying too much about typing in those characters or escaping them would be to try to wildcard them instead. In your case try something along these lines (use a more specific pattern if there are other files matching this one besides the 2 files of interest):

aws s3 sync R*glement*pdf s3://bucket_name/raw_files/ --sse


Where to obtain the token to register a GitLab runner? I have a GitLab repository.
I want to add a runner to it.
I was able to install it, with the instruction here without Docker (step 3).

Now I have to register it according to the instructions here. In the beggining of that file it is wrriten that I have to obtain a token, but I don’t understand where do I obtain the token from.

In this link it is wrriten that
I can get the token from CI/CD -> settings
but CI/CD does not have a settings option.

Also, in the registration instruction in step 2, it is written:
Please enter the gitlab-ci coordinator URL but there is no documentation anywhere about what is this URL

For a specific GitLab runner tied to a project, the token can be found in Step 3; in Project -> Settings -> CI/CD -> Runners in the Setup a specific Runner manually section.

You will also be able to find the URL which you need to use to register the runner in Step 2.


Dockerize a windows GUI application Is it possible to run a Windows GUI executable (notepad.exe for example) into a container ?
I want to do this because i do not trust some software i install on my personal computer and i want to be sure they are runned into a sandbox
Thanks

I'd have to say that I don't think it's possible and if you understood windows containers you really wouldn't want to do it anyway. Last time I checked, a windows container weighs in at gigabytes++ and that's huge to run notepad.
You can run xwindows apps in a container, but windows is, basically, complete shite in a container.
If you want a sandbox then you should checkout VirtualBox or Hyper-V. You can script either with Hashicorp Vagrant.


What steps could be taken to inherit and override some parts of a Chef cookbook? With 3200+ cookbooks available in Chef supermarket, often there is already a cookbook that covers the need I need to solve.

Almost every time there is some change required to a template, or a variable, or a resource in an existing cookbook.

What are the steps to override and/or improve existing cookbooks? Without forking their code, and retaining the ability to get version updates in the future.

There are a lot of specifics but the overall pattern we use is "wrap and extend". The general idea is to make a cookbook that depends on the community cookbook, usually named mycompany_originalthing, and then make recipes in that which call include_recipe 'originalthing::whatever' but with either more stuff added before/after or with calls to things like edit_resource to change resources. Avoid edit_resource when possible since it leads to brittle code, but it is there if you need it. You can also use wrapper cookbooks to set attributes, subclass or wrap custom resources, and so on.

For the specific case of "I need to tweak a template in a community recipe" it would look like this:

include_recipe 'original::whatever'

edit_resource!(:template, '/path/to/something') do
  source 'mytemplate.erb'
  cookbook 'mycompany_original'
end

You can find more details about edit_resource and friends at https://coderanger.net/rewind/

With cookbooks based around custom resources instead of recipes, things can get more complex but the specifics depend a lot on the how exactly the cookbook you are extending is written.


When setting up HashiCorp Vault in HA mode, is it normal that you have to unseal each Vault node? My goal has been to setup Vault in HA mode.

Based on the docs, I was under the assumption that if you had a node that is in 'standby' it would forward to the active node, thus not having to unseal (assuming the active node was already unsealed). Right now, non of the nodes forward/redirect - they just serve the Vault UI.

Here's my existing Vault config:

ui = true
api_addr = "https://{{inventory_hostname}}:8200"
cluster_addr = "https://{{inventory_hostname}}:8201"

storage "etcd" {
  address  = "https://etcdhost:2379"
  etcd_api = "v3"
  ha_enabled = "true"
  tls_ca_file = "{{tls_path}}rootCA.crt"
  tls_cert_file = "{{tls_path}}etcd.crt"
  tls_key_file = "{{tls_path}}etcd.key"
  path = "{{etcd_path}}"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file="{{tls_path}}etcd.crt"
  tls_key_file="{{tls_path}}etcd.key"
}

According to this documentation, each vault has to be unsealed in a HA setup.

  Now you need to initializing and unsealing each Vault instance.

# Initialize vault_s1
$ vault operator init

# Unseal the vault_s1
$ vault operator unseal <unseal_key_1>

$ vault operator unseal <unseal_key_2>

$ vault operator unseal <unseal_key_3>


Travis tests for R package fail with little explanation I am writing an R
package and using
Travis for Continuous Integration. Tests and R CMD CHECK pass locally and for
3 versions of R under Linux (oldrel, release, and devel). All three versions
fail under macOS with no information in the Travis logs:

...
R CMD check fail logs
0.01s$ for name in $(find "${RCHECK_DIR}" -type f -name "*fail");do echo ">>> Filename: ${name} <<<";cat ${name};done
log
R CMD check log logs
...

I checked in the Travis logs that all the tests I wrote in testthat fail. The Travis tests once passed for both Linux and macOS, so I could dissect the code changes to detect the issue, but ideally I could use something more direct and faster than groping in the dark waiting for an answer from Travis.

Any ideas?

If you are using the recommended setting for R packages of upgrading warnings to errors, i.e. with this line in .travis.yml:

warnings_are_errors: true

then the place to look is the previous heading in the Travis log, called Checking package.

Example of problem with missing font

In my case, the Travis log under the heading Checking package showed that the
inconsolata font was missing, so the vignettes failed to build:

* checking PDF version of manual ... WARNING
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
LaTeX errors found:
! LaTeX Error: File `inconsolata.sty' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)
! Emergency stop.
<read *>

l.276 ^^M

!  ==> Fatal error occurred, no output PDF file produced!
* checking PDF version of manual without hyperrefs or index ... ERROR
* DONE
Status: 1 ERROR, 1 WARNING

This caused an error for LaTeX, a warning for R, and an error for Travis.

Solution to problem

In my case, the problem was due to Travis using TeXLive 2019 and the CTAN
mirrors having yet to update to 2019.

The issue appears in the Travis R community forum:

  might be related to the recent switch to TexLive-2019 as the LaTeX package manager seems to fail finding inconsolata in the repository.

  ...

  It looks to me like the mirror chosen was has not updated to 2019, so I think this will resolve itself naturally when the mirrors catch up. If not I can look into it.

In the meantime, I solved it by forcing a previous version of TeXLive with a downgrade to OSX 10.12 in .travis.yml:

# macOS 10.12 version
osx_image: xcode9.2

See this Travis
page for the
correspondence between XCode and OSX versions and the original thread in DevOps
StackExchange.


How to set a shared-library globally using the JCAC plugin? This Q&A is related to this one and is slightly different.

https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/demos/jenkins/jenkins.yaml

It should be possible by configuring it like this:

unclassified:
  globalLibraries:
    libraries:
      - name: "some-lib"
        retriever:
          modernSCM:
            scm:
              git:
                remote: "https://github.com/some-project.git"
                credentialsId: 'some-credentials'


View all AWS S3 buckets and list each buckets storage used I have multiple AWS accounts and I need to list all S3 buckets per account and then view each buckets total size.

Currently, I can only view the storage size of a single S3 bucket with:

aws s3 ls s3://mybucket --recursive --human-readable --summarize

Resolution 1

So I solved this with the following script. I originally posted the question just in case there was an easier way that I was not aware of.

#!/bin/bash
aws_profile=('profile1' 'profile2' 'profile3');

#loop AWS profiles
for i in "${aws_profile[@]}"; do
  echo "${i}"
  buckets=($(aws --profile "${i}" --region your_region s3 ls s3:// --recursive | awk '{print $3}'))

  #loop S3 buckets
  for j in "${buckets[@]}"; do
  echo "${j}"
  aws --profile "${i}" --region your_region s3 ls s3://"${j}" --recursive --human-readable --summarize | awk END'{print}'
  done

done

Resolution 2

Using Dashboards in CloudWatch in the AWS console.

You can then simply specify all S3 buckets and add the numbers stats to show the storage size metrics.

This won't cost you plenty of API calls and can be significantly faster depending on the size of the s3 buckets(takes quite awhile to get the size on very large buckets).

Verdict

Creating the Dashboard (Resolution 2) on each AWS account was the most efficient option for me cause it is way quicker for me to log in and grab the metrics manually from each AWS account than to wait for the scripts API calls to finish. :(


How to manage Consul and its quorum in auto-scaling environments? We have auto-scaling Docker environments in which we use Consul for service discovery. These environments can add or remove one instance every few minutes.

Our early Consul testing showed that it was very easy for Consul to loose its quorum. Perhaps naively, our very first experimentation was a setup in which we would start a Consul server on all instances and have that Consul server join the cluster. That part was working fine.

However, Consul does not reap unreachable nodes quickly (it take about 72 hours?) in a very scalable environments that means that the list of Consul servers keeps growing and over time, most of them are "unreachable" and at that point, the cluster loses its quorum.

We've seen armon's response from almost two years ago on this issue on GitHub: https://github.com/hashicorp/consul/issues/454#issuecomment-125767550

  Most of these problems are caused by our default behavior of
  attempting a graceful leave. Our mental model is that servers are long
  lived and don't shutdown for any reason other than unexpected power
  loss, or a graceful maintenance in which case you need to leave the
  cluster. In retrospect that was a bad default. Almost all of this can
  be avoided by just kill -9 the Consul server, in affect simulating
  power loss.

We were trying to avoid running dedicated, long-lived nodes. Keep in mind that at no point, we remove N/2+1 instances from an auto-scaling group. The EC2 cluster is able at any point in time to reach most of the nodes and should be able to vote whether a node should be removed from the Consul (or other tool) cluster.

I would set the leave_on_terminate option to true. As per the documentation

  leave_on_terminate If enabled, when the agent receives a TERM signal,
  it will send a Leave message to the rest of the cluster and gracefully
  leave. The default behavior for this feature varies based on whether
  or not the agent is running as a client or a server (prior to Consul
  0.7 the default value was unconditionally set to false). On agents in client-mode, this defaults to true and for agents in server-mode, this
  defaults to false.

What happens when a node is shutdown gracefully is to send SIGTERM to all processes before shutdown, with this setting on the consul agent will leave the cluster so it won't be considered as a node which can restart and be back in the cluster in a few hours (which is what your quote says it does by default).


Which system user should I use to run Postgres? I am using a postgres docker image configured with the following environment:

    image: healthcheck/postgres:alpine
    environment:
      POSTGRES_USER: postgres

I have the following questions regarding this setup:

What is the default user of postgresql?
How does hardcoding the postgres user improve security?
What special rights does the postgres user have?
What is the upside and downside of using an unprivileged user ?

What is the default user of postgresql?

postgres

  How does hardcoding the postgres user improve security?

It's not using the root user.

You can use any account you like, but consistency is a Wonderful Thing, especially when you have lots [and lots] of Postgres clusters to manage.

  What special rights does the postgres user have?

Not many.

It installs itself a hole through the local firewall but, otherwise, it's just another user.

  What is the upside and downside of using an unprivileged user ?

You should never run applications (or databases) as root (unless they actually need to be).

Always keep the biggest and best tools in the toolbox for yourself, ready to sort out the mess that Applications (or Users) make.  If you run stuff as root and it decides to do something really stupid to your machine, you may not be able to fix it.


Do I have to include gitlab variables in my terraform code in order for my credentials to be recognized? I am trying to set up a pipeline, which is supposed to deploy terraform code into an aws account. The code works fine locally, and I have setup and imported the state bucket and dynamodb table which enables state locking.
However, when I run the code in a gitlab ci, I get the following error message:

Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.
│
│ Please see https://www.terraform.io/docs/language/settings/backends/s3.html
│ for more information about providing credentials.
│
│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.
│   For verbose messaging see aws.Config.CredentialsChainVerboseErrors

I export the aws credentials as variables in the before_script like so:
before_script:
  - rm -rf .terraform
  - terraform --version
  - export AWS_ACCESS_KEY_ID
  - export AWS_ROLE_ARN
  - export AWS_DEFAULT_REGION
  - export AWS_ROLE_ARN

Is there another step that I am missing, or do I maybe have to include the credential variables in the terraform code?

The answer to this question is no. I merely forgot to export the AWS_SECRET_ACCESS_KEY variable in the before_script section.


File transfer from Application server to Db Server without a shared folder access but using Service Account I am working as a Business Analyst for an IT firm. My team has stumbled upon a certain feature where they are required to use Service account.
We are working for a Bank and they have strong policies of keeping no Shared folder on Db server, for this they have advised us and provided us Service accounts with help of their server team.
Naturally my team is unaware on how to use this Service account for transferring the file From App server A to Db server B and thus i will be needing help from you guys.

If you ask me i dont know exactly what a service account is, but judging by the name it is a credentials which allows access within different servers.
Can anyone advice on how to use this for file tranfer?

There is a general lack of understanding here.  I suspect your company doesn't understand your "requirement" and the Server team has provided something that either won't do the job or your team don't know how to make it work.

You say that you "need" to get a file onto a Database server - my first question would be - "why"?

What's in this file that you need to put it anywhere near the Database server?

Your company has a Policy against doing this through file shares on the Database server.  [Being a DBA] I completely understand (and agree) with this Policy.

On its own, a Service Account isn't going to be much use to you.
You might be able to use it with some File Transfer utility to copy the file but, again, why do you want to do this?

Take a step back and describe the "Big Picture" problem that you're trying to solve here.


How to test a cluster of microservices? I took a look into Server Spec and it says it clearly:

  Remark: serverspec test suites are meant to be run against a single machine  (or docker container). In other words, you should not try to issue a single rspec command that would harvest and run tests against multiple machines or containers. You need to issue one rspec command for each of them.

So I can do test per-server, which is good for some cases. But the problem is the following: my microservice architecture has a auto-discovery service, so some services are known after querying it. Is there any project to express this? I know I can tool around with ruby (or python if I choose  testinfra or other)

Something integrates with Consul with serverspec or similar will be great, in 2014 people were looking for this, does anyone know of any project that tackled this issue?

If nothing exists to easily mock Consul, have you considered spinning up a Consul service along with your other microservices? Consul can run in Docker, so this should be fairly straightforward if you use docker-compose and your apps can be configured to point to a different Consul endpoint.

Once your docker-compose is configured to spin up all the infrastructure/microservices required for your test, you will be able to use whatever tools you're familiar with to test your microservices, e.g. Selenium.


what's the final output image in multistage build? Below is the dockerfile that creates an image:
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt

FROM diamol/base AS test-stage
COPY --from=build-stage /build.txt /build.txt
RUN echo 'Testing...' >> /build.txt

FROM diamol/base
COPY --from=test-stage /build.txt /build.txt
RUN echo 'Done...' >> /build.txt

and my understanding is, the final image consists layers of diamol/base and the layer created after the last RUN command as picture below shows:

Is my understanding correct?

The final image will have all the layers from diamol/base as well as two additional layers - one for COPY and one for RUN command.
File /build.txt will have the following contents:
Building...
Testing...
Done...


Problem with Ansible Asynchronous actions I'm trying to run two simple tasks
Task A starts an Ipref3 server and waits for a client to connect
Task B starts to run from host-1 until the given timeout
once Task B (host-1) is over another Task B run from host-2
Task B starts to run from host-2 until the given timeout
What is a good approach to orchestrating this use-case?
This is what I tried
Task B runs the client from multiple hosts (host-1, host-2, ..)
- name: Run iperf3 client
  win_command: iperf3 -c {{ iperf_server }} -t {{ iperf_time }}
  args:
    chdir: "{{ iperf_dest }}"
  register: iperf3_out

- debug:
    var: iperf3_out.stdout

Task A run the server
- name: Start iperf3 server
  shell: iperf3 --server
  async: "{{ iperf_async }}" # iperf_async = iperf_time + 10
  poll: "{{ iperf_poll }}"   # iperf_poll  = 0

Is there a way to know when to confirm Task A is finished before I move to the next task.
NOTE: Order of execution is important Server (Task A) needs to run before the client (Task B)

Order of execution is important Server (Task A) needs to run before the client (Task B)

Then why are your running Task A async in that case ? Is it just because iperf --server is not falling in the background and dying as soon as the shell task finishes ? A quick look at the documentation shows we can probably work around that problem with something like:
iperf3 --server --daemon --pidfile /path/to/iperf3.pid --logfile /path/to/log

Notes:

I added the logfile option since I guess it can be useful but it is not mandatory for the current issue
I took for granted you are running the server on a linux platform. Running it on Windows would apparently slightly change the below scenario. See Running as a service on windows

From what I understand the last unsolved issue is running the client tasks sequentially on each client host. This can be done using the serial keyword on your play.
Here is an untested pseudo ansible setup summarizing what I understand from your requirements. Adapt to your real use-case.
inventory/iperf_hosts.yml
---
iperf_servers:
  vars:
    iperf_pid_file: /tmp/iperf.pid
    iperf_log_file: /tmp/iperf.log
  hosts:
    server1.local:
iperf_clients:
  vars:
    iperf_time: 1
    iperf_dest: some\win\directory
  hosts:
    host1.local:
    host2.local:
    host3.local:

playbook.yml
---
- name: Start the iperf server
  hosts: iperf_servers

  tasks:
    - name: Fire the start command
      command: "iperf3 --server --daemon --pidfile {{ iperf_pid_file }} --logfile {{ iperf_log_file }}"

- name: Launch clients one after the other.
  hosts: iperf_clients
  serial: 1

  tasks:
    - name: Run iperf3 client
      win_command: "iperf3 -c {{ hostvars[group['iperf_servers'][0]].inventory_hostname }} -t {{ iperf_time }}"
      args:
        chdir: "{{ iperf_dest }}"
      register: iperf3_out

    - debug:
        var: iperf3_out.stdout

- name: Now we are over with clients, kill the server
  hosts: iperf_servers
  # We already gathered facts at first play so we disable here
  # You might want to disable in all other plays if you don't need facts at all
  gather_facts: false

  tasks:
    - name: Slurp pid from file
      slurp:
        path: "{{ iperf_pid_file }}"
      register: slurped_pid

    - name: kill the iperf3 process
      command: "kill {{ slurped_pid.content | b64decode }}"


How to prohibit access to internals of Docker container? I want to deliver my app to customers in form of docker image. But it is crucial to ensure that end-user do not alter anything inside the container. User should only be able to run/stop the container and interact with the container via network.

Is it possible to prohibit access to internals of container?
Is it possible to verify integrity of image that container made from?

In short, you cannot prevent your customers from modifying containers they run in their own infrastructure.  Containers are not like binaries that can be obfuscated; they are runtime environments.  The code you distribute inside the container, however, can be obfuscated.

Your question alludes to a third-party support issue: clients modifying software they run in their own environments.  If you provide tools to operate the containers you supply (e.g. monitoring and logging) then the clients should agree (as part of a software license) not to make unauthorized modifications to them.  This applies to all types of third-party software, not just containers.

Depending on your situation, you may also have the option to provide your app as Software As A Service (SaaS) running in a cloud infrastructure.

If your client requires your containers be run on their infrastructure, and refuses to abide by modification restrictions, then you probably don't want to try to support their use of your software.


Chef syntax with "do" statement While reviewing a chef.io whitepaper, I encountered the following syntax:

# Load your AWS credentials databag
include_recipe ‘aws’
aws = data_bag_item(‘aws’, ‘main’)
aws_elastic_lb ‘elb_qa’ do
  aws_access_key aws[‘aws_access_key_id’]
  aws_secret_access_key aws[‘aws_secret_access_key’]
  name ‘elb_qa’
  action :register
end

Can you explain or refer to documentation about the "do" statement?

The common example for the Ruby "do" statement is using "each", which cannot be found in the example above. What does the syntax mean? what here is a parameter, variable, assignment, function, etc.. ?

what you're looking at is a resource declaration, mostly a class taking a block as parameter.

In ruby the do code end syntax is the prefered way over the curly braces syntax { code } which is prefered for inline code.

So mostly, what you have is the same as:

aws_elastic_lb('elb_qa'){ aws_access_key aws[‘aws_access_key_id’]; aws_secret_access_key aws[‘aws_secret_access_key’]; name ‘elb_qa’; action :register }

In recent version of chef (14.X) the name attribute should be the name of the block (what is just after the resource type (aws_elastic_lb)

More details in this SO quastion about ruby syntax.

Concerning Chef itself I highly suggest you follow the tutorials at https://learn.chef.io for a better overview of what is what in cookbooks.


In k8s, do the controller nodes have to be in same VPC as the worker nodes for a cluster? I'm new to k8s and trying to grasp the infrastructure concepts, and trying to apply them in AWS. I know a k8s cluster has a control plane, which has the management nodes, and a data plane, which has the worker nodes. Do they have to run on their own respective VPC, or can they run on the same VPC? I hope that makes sense.

For Amazon EKS, the control plane is running in a separate VPC that is managed by Amazon. You only need to care about the VPC for the worker nodes.

Amazon EKS control plane – Deployed and managed by Amazon EKS in an Amazon EKS managed VPC. When you create the cluster, Amazon EKS creates and manages network interfaces in your account that have Amazon EKS  in their description. These network interfaces allow AWS Fargate and Amazon EC2 instances to communicate with the control plane.

From Amazon EKS Networking


DNS server with api access Experts, i have a requirement and wanted to find a tool that will be helpful for me.

We have a inhouse tool, that will help developers request/lease a VM for a short duration of time. When the VM is assigned, i send the developer an email with details of the VM and the application details. I currently use xip.io to send a fully qualified DNS name. Something like this

https://omni.54.84.86.26.xip.io

https://auth.54.84.86.26.xip.io

https://mgmt.54.84.86.26.xip.io

This works well, but the feedback that i got from my developers is ,

The url is very hard to remember

Everytime is request for a VM i get a new URL and i cant bookmark it. it keeps changing everytime.

I think their concern is valid. One idea to solve this is to create a per defined DNS names with username in it like

omni-prajan.myorg.com

admin-prajan.myorg.com

mgmt-prajan.myorg.com

where prajan is the username. So everytime i request for a vm, i need to programmatically modify IP that is associated with this domain name in my dns server. We are using AWS now.

Is there a better way to handle this requirement?

Route53 has an API for creating DNS records.  The API is a bit obtuse, but there are a plethora of wrappers for various languages.


Tomcat deploy war with Ant - error writing to server I am using Ant to deploy a war to a Tomcat server, but it gives me the following error:

    BUILD FAILED
    /home/deploy/Documents/Work/Deploy_Wars/xrepo/deployment/build.xml:37: The following error occurred while executing this line:
    /home/deploy/Documents/Work/Deploy_Wars/xrepo/deployment/ant_scripts/deploy_wars.xml:36: The following error occurred while executing this line:
    /home/deploy/Documents/Work/Deploy_Wars/xrepo/deployment/ant_scripts/deploy_wars.xml:42: java.io.IOException: Error writing to server
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:699)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:711)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1585)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at org.apache.catalina.ant.AbstractCatalinaTask.execute(AbstractCatalinaTask.java:223)
        at org.apache.catalina.ant.DeployTask.execute(DeployTask.java:201)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:350)
        at org.apache.tools.ant.Target.execute(Target.java:448)
        at org.apache.tools.ant.Target.performTasks(Target.java:469)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
        at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:36)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
        at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:446)
        at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

Here is the Ant target that I use for this deployment (and here is the line where the exception is thrown):

    <target name="deploy-debug" description="Install DEBUG">
        <deploy url="http://debug.webiste.ro:8180/manager" username="admin" password="password"
                path="/debug-war" war="wars/debug-war.war"/>
    </target>

I am using Tomcat 9.0.10 and Ant 1.10.3 on Ubuntu 18.04.

I saw some people suggesting that the tomcat user is not properly defined in tomcat_users.xml. I have modified it, but it still gives me the exception when I try to deploy. I can succsefully login on the Tomcat manager app in the browser. Here is my tomcat_users.xml:

    <tomcat-users xmlns="http://tomcat.apache.org/xml"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://tomcat.apache.org/xml tomcat-users.xsd"
          version="1.0">
          <role rolename="tomcat"/>
          <role rolename="manager-script"/>
          <role rolename="manager-gui"/>
          <role rolename="admin-gui"/>
          <role rolename="admin-script"/>
          <user username="admin" password="password" roles="manager-gui,manager-script,admin-gui,admin-script"/>
    </tomcat-users>

The war file that I am trying to deploy has 80 Mb which is bigger than the default maximum upload value of Tomcat (50 Mb). I have tried increasing the maximum upload value to 100 Mb but it still gives me the Exception.

Any ideas on this problem?

It could be disk space, or hot deploy. For hot deploy, you can put the following parameters:

...
<target name="tomcat-stop">
    <exec executable="${server.home}/bin/catalina.bat">
        <arg value="stop"/>
    </exec>
</target>

<target name="tomcat-start">
    <exec executable="${server.home}/bin/startup.bat">
        <arg value="start"/>
    </exec>
</target>
...
<target name="all" depends="tomcat-stop,clean,init,compile,junit-slow,make_war,deploy,tomcat-start"></target>

The answer is at this link: https://stackoverflow.com/a/32482795/8442153


AWS Temporary Permission Granting Tool for Account/Service "Firefights" I'm looking for a service or third party solution that allows for granting temporary access to an IAM user (or group) to one or more services in an AWS account.
The example I have is there is an outage in a production account, and I would like to grant temporary access to developers/sys admins to resolve the issue. Under normal circumstances, they would not have access to this particular service/account.
Ideally, I would like this to me a managed off the shelf solution vs something like STS, which would require quite a bit of configuration. Example, a system outage occurs for unknown reasons in a production account that the on call SRE does not normally have access to. A manager or lead should quickly be able to temporarily escalate privileges for the user until the "firefight" is resolved.

Create an IAM role in prod with all the permissions you'd like the devs to have.
When you want devs to have prod access, add the sts:AssumeRole permission to their IAM group.
When it's time to revoke access, remove the sts:AssumeRole permission.

More details in my blog article here.


How to Versionize Ansible (Tower) with GitHUB I have configured Ansible Tower with Centos 7.2 and I am able to see my Tower dashboard.  My major concern is how to do versioninzation in Ansible Tower as given in this link with Github.

Since I have written the playbook for configuration changes in my Centos server and push my changes to GITHUB in a branch as SAAS_1 now that I have to integrate this changes to Ansible Tower, how can I see this versioning? How can I visually see all of this content in Ansible Tower?

Like production -1.1 UFT-2.1 and continuation based on changes enabled

one strategy is to put your playbook repo(s) thru a regular build/deploy pipeline, where you separate between "nonrelease" and "release", for example by linking the master branch to "release" (most CI tools support this out of the box). Then you'd have two "streams" of artifacts you could deploy, one for release and one for non-release (or release candidate or whatever you'd like to call it). These can be auto-deployed to different folders inside Ansible tower and then picked up as different "projects". That way youd have two "instances" of every playbook folder (or "project" as Tower calls them).


Sub-subdomain Domain name for different environment If we have two components in a API

api front-end

and we have three environments staging qa development

and a domain name superproduct.com

What would be the most logical domain spacing:

http://development.front-end.superproduct.com
http://development.api.superproduct.com

http://qa.front-end.superproduct.com
http://qa.api.superproduct.com

http://staging.front-end.superproduct.com
http://staging.api.superproduct.com

or the other way around:

http://front-end.development.superproduct.com
http://api.development.superproduct.com

http://front-end.qa.superproduct.com
http://api.qa.superproduct.com

http://front-end.staging.superproduct.com
http://api.staging.superproduct.com

Both are completely correct. It can differ organisation by organisation.
Also, once you start using one of the domain namespace type, you will gradually remember the format and no issues will come. However, I can see one issue here which is related to SSL.

If you go with any one of the above mentioned namespaces, you need to generate multiple SSL certificates such as *.api.superproduct.com and *.front-end.superproduct.com.

If you go with other namespace type then SSL certs will be required for *.staging.superproduct.com, *.qa.superproduct.com and *.development.superproduct.com.

To avoid this, I would suggest you to have the domains in either of the following formats.

<component-name>-<env-name>.superproduct.com

Examples are:
api-qa.superproduct.com, api-staging.superproduct.com etc.

<env-name>-<component-name>.superproduct.com

Examples are:
qa-api.superproduct.com, staging-api.superproduct.com etc.

In this case, only one SSL certificate for *.superproduct.com will be required.

You can go ahead with the best possible option as per your requirements and analysis.


Dockerized Flask: Connection reset by peer Trying to connect to a dockerized Flask app fails with error 104, 'Connection reset by peer' using this minimal example:

app.py:

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

Dockerfile:

FROM python:alpine
RUN pip install flask
COPY . /src/
EXPOSE 5000
ENTRYPOINT ["python", "/src/app.py"]

docker-compose.yml:

…
test:
    build: .
    ports:
        - 127.0.0.1:5000:5000

Flask app seems to be running as expected:

$ docker logs test
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Trying to connect from outside fails:

$ http http://127.0.0.1:5000/
http: error: ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) while doing GET request to URL: http://127.0.0.1:5000/

Any ideas, why I can't get to see "Hello World!" here?

Trying to connect from outside fails

Are you actually connecting from outside? Flask is binding to localhost (127.0.0.1) and that will only be reachable from within the container. If you're on your local machine, you'll need flask to bind to all IP's:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run(host='0.0.0.0')

That works for me successfully.


Import data from SharePoint to Azure DevOps We have migrated to DevOps from SharePoint 2010. We need to Import all the user stories we have in sharepoint to Azure DevOps . Can any one help me with a better way to go about this?
Know someone who can answer?

There are no tools to import data from SharePoint to Azure DevOps. You can consider the following:

Export SharePoint List to csv and then import it to Azure DevOps
Use Excel integration. Export Sharepoint list to Excel and then copy/paste data to Excel linked with Azure DevOps and publish it.


Git - Everyone Pushing to Same Branch in Different Files Database team is implementing code changes, using Visual Studio SSDT database projects with Git source control. Everyone is pushing to main Release branch with code review (only 5 developers on the team). All the database coworkers are only allowed pushing to different files only (tables, sprocs, functions), etc. The way work is assigned, none of us push or work the same sql file. Eventually all good changes from Release (currently in Work) are merged into Master branch (Production Ready).

Code Review ---> Push to Release Branch (Currently in work during Sprint) ---> Merge to Master Production Ready Branch

(a) What are the negative consequences of utilizing this strategy in Git?

(b) For cleaner history, should everyone Rebase ReleasePublic Remote into ReleaseLocal, or conducting Pull? (Fetch/Merge), I would think Rebase is answer for cleaner history.

Note: I agree, it would be annoying only if we are working on the same file and pushing changes. Alternative Strategy to create different feature branches and then merge into main branch. We are refraining from this strategy since each developer has 10 dba admin related changes a day, creating many branches and merges is time consuming and cumbersome-

https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow

This is not a "strategy". This is a sure way toward chaos.

The idea behind "trunk branch", which means having one branch for integration, does not mean that everyone should be pushing their changes to this branch directly. It only means that his branch is the place where integration happens.

Why do you even need two branches, Release and Master? What benefit does this bring? Probably just the benefit of confusing git history into an incomprehensible mess so that no one knows what is happening.

Stop this madness!

Make each change on its own pull-request branch (call it feature branches if you like).
Make sure that these pull-requests are not merged into the main integration branch before the code there goes through some minimal code-review.
Restrict from pushing directly to your integration branch.
For cleaner history, make sure that the pull-request branch if first rebased on top of integration.
Make sure that merges of pull-requests are not fast-forwarded and clearly visible in history.

The above will allow for several benefits,

Higher quality and less "human error" mistakes.
Learning and teaching each other cool dba tricks (during code reviews).
Keeping the integration branch green most of the time (it works!).
High trust that you are not shooting yourself in the leg using your process.
Way less "oops, I overwrote your change" kind of merges.
High trust that new team members are not going to mess things up.
Git history that is a thing of beauty and a joy forever.

And more ...

Prettier explanation of the "Feature Branch Workflow" is available on the Atlassian Git website here https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow


How to connect to external etcd server inside of docker container (python application)? I have a Cement application that runs inside of a docker container. In this image I have to connect to etcd. Inside of my docker image I have config that for etcd is set to localhost that I know is not correct!

On my local machine etcd is up and running:

tcp4       0      0  localhost.2379         localhost.51949        ESTABLISHED

The question is how should I connect to etcd server which is outside of docker container? etcd is installed normally without using docker.

Output of my python application when I run it with docker run -p 8084:8084 my_python_image:

Traceback (most recent call last):
  File "my_app.py", line 64, in <module>
    app.run()
  File "/usr/local/lib/python2.7/site-packages/cement/core/foundation.py", line 882, in run
    return_val = self.controller._dispatch()
  File "/usr/local/lib/python2.7/site-packages/cement/core/controller.py", line 477, in _dispatch
    return func()
  File "my_app.py", line 45, in default
    module_rpc = rpc.Rpc('my_module', params=app.config.get_section_dict('etcd'))
  File "/usr/local/lib/python2.7/site-packages/rpc_server/rpc.py", line 31, in __init__
    self.etcd_client.write(self.etcd_nodes, None, dir=True)
  File "/usr/local/lib/python2.7/site-packages/etcd/client.py", line 500, in write
    response = self.api_execute(path, method, params=params)
  File "/usr/local/lib/python2.7/site-packages/etcd/client.py", line 893, in wrapper
    cause=e
etcd.EtcdConnectionFailed: Connection to etcd failed due to MaxRetryError(u"HTTPConnectionPool(host=u'localhost', port=2379): Max retries exceeded with url: /v2/keys/my_module/nodes (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcc33eaa690>: Failed to establish a new connection: [Errno 111] Connection refused',))",)

The error you see in your container is:

  etcd.EtcdConnectionFailed: Connection to etcd failed due to MaxRetryError(u"HTTPConnectionPool(host=u'localhost', port=2379): Max retries exceeded with url: /v2/keys/my_module/nodes (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))",)

If you take a close look there, you can see that it's trying to connect to localhost.  Inside your container, localhost is means "the current container" (much like localhost on your host means "the current host"). If you want to connect to an etcd instance outside of the container, you'll need to use the ip address (or hostname) of another server.

If you're running (a) running Docker locally, (b) etcd on your host is listening on all addresses, and (c) there are no firewall rules that would otherwise prohibit the connection, then you can use the address of the Docker bridge associated with your container as the address of your host.  This will be the default gateway as seen from inside the container.  You can extract the address with something like:

ip route show | awk '$1 == "default" {print $3}'


Linking nodes in a cluster to run a single instance of a docker container In this cluster, there are 11 identical nodes, each with 16gb of memory. So far, I've been able to connect each node of the cluster to a docker swarm hosted on the control node. My end goal is to run a single instance of a container that utilizes the resources of each node so the total memory of that container would be 176gb. How should I accomplish this?

Each worker node is running Ubuntu Server 20.04 and the control node is running Ubuntu Desktop 16.04, all with docker engine 19.03.8.

This is not really possible with container orchestration. Containers are bound by host and cannot be larger than host machines.

At the same time, depending on your specific tasks there may exist specialized distributed software that may be suitable for your needs - i.e. Apache Spark covers a lot of computational use cases.


Should compilers be kept in artifactory? We are setting up our source and build system in AWS.  we will have two persistent VMs hosting source and artifactory.  We have a bunch of compilers (a few Gigs) we want available via NFS, is this something that should be hosted in artifactory?

I'm looking for the 'best practice' answer.  I can't find much info on the thought behind this so any links would be appreciated.

There are a few main reasons why you would use Artifactory (or any other binary repository manager) over a traditional file storage (in your case NFS).

The ability to manage your artifact versions
A central location to access artifacts across infrastructure (which you have accomplished with NFS)
The ability to download/recreate previous versions of the artifacts (you could also use a git/svn repository for this, but it would be contain MUCH more irrelevant metadata about the binary artifacts than would be useful to a user)
You want to control who/what has permissions to access the stored artifacts

If none of these are applicable in your use-case, I would say that Artifactory would be overkill and the effort to create and maintain it would not out way the value you would receive from it. Although I have used it in the past, and it is a great tool for what it aims to accomplish :)


Updating Opera browser through command line (Windows) I'm writing a script that suppose to, among others, update Opera browser on the machine it's running on. Is there a way to update it through cmd? Is there a universal way to do so for any chromium based browser?

One you use chocolatey to install Opera. Upgrading is also supported:

choco upgrade opera


Kubernetes: CPU Limits at (200%)? So, let's say there's a cluster node, with 4 CPU cores.

It's running two pods.
Each pod sets the CPU requirement at: 500m, but the CPU limit at 4000m.

running kubectl describe node <noode> tells me CPU limit is at 8 (200%)

I understand it claims 200% because I have two pods, each setting its CPU limit at 4 CPU cores.

Is that just semantics or is there a potential issue? I mean, if both pods need to burst to 3 CPU cores (below each one's limit) it would not be physically possible, and they'd have to settle with less.

Is there a potential issue with the sum of all pod limits, exceeding their host node CPU capability? or is just an indication of what's been asked as CPU limit, knowing it may not be able to serve that -- after all, pods may not be aware of each other being in the same node.

This works exactly in the way you're describing.  You have two pods which each have a limit of 4 CPUs scheduled on the same node; so the total of the limits on that node is 8 CPUs; which is 200% of the node's physical capacity; and if the two nodes combined go over the node's physical abilities then the host kernel will time-slice available resources.  For CPU there's nothing especially wrong with this.

The exact same presentation in the kubectl describe node output applies to memory, and there is the potential for trouble with that: if your pod limits add up to 200% of a node's physical memory, and together they exceed the physical capacity, then the host kernel will respond by killing one of them off.  Lower requests will allow more things to be potentially scheduled on the same node, but keeping requests closer to limits gives you a better chance of them staying running long-term.


packge.json not found while executing on Jenkins My Dockerfile looks something like this.

FROM mhart/alpine-node:8.11.3

RUN mkdir -p /app
COPY ./ /app

WORKDIR /app/build

RUN yarn global add serve
CMD ["serve", "-l", "3000"]

EXPOSE 3000

And then JenkinsFile looks something like this.

 node {
  try {

    stage('Checkout source code') {
        checkout scm
    }

    stage('Install packages') {
      sh("docker run --rm -v `pwd`:/app -w /app node yarn install")
      //sh("sudo chown -R jenkins: ./node_modules")
    }

    stage('Set the enviroment variables') {
      sh("echo set-env-variables")
      }

    stage('Build static assets') {

      sh("docker run --rm -v `pwd`:/app -w /app node yarn build")
    }
  }}

When I do run it on Jenkins,the console output says error Couldn't find a package.json file in "/app" and also it gives an error sudo not found even though I have added Jenkins ALL=(ALL) NOPASSWD: ALL to /etc/sudoers file.

I run the commands listed in my Jenkinsfile on my terminal and they all work fine but when i run them on Jenkins , they dont work.

Because you are setting the container's WORKDIR to /app/build so your package.json is not in the working directory, but in the parent of the working directory.


What is the duration for the App Engine task queue tombstones? Recently I got TombstonedTaskError and I would like to know what is the duration for the TombstonedTaskError?

TombstonedTaskError is raised if you're attempting to enqueue a named task using a name that has been used recently, see What is TombstonedTaskError from App Engine's Task Queue?.

The duration of the de-duplication logic has been documented since that answer was posted. From Naming a task (emphasis mine):

  When you create a new task, App Engine assigns the task a unique name
  by default. However, you can assign your own name to a task by using
  the name parameter. An advantage of assigning your own task names is
  that named tasks are de-duplicated, which means you can use task names
  to guarantee that a task is only added once. De-duplication
  continues for 9 days after the task is completed or deleted.


Does anyone have resources, tools, ideas, for automating database create, update, modify, delete I've been tasked to find a way to automate promotion of database creation, changes and other modification related to application changes in a devops world.
My databases include MS SQL, Oracle, MySQL and could include NoSQL databases.
I've been doing what I can with web searches and all I seem to be able to find are powershell maintenance scripts (backups, rebuild indexes, etc). Nothing really about taking DDL out of a VCS like subversion, using jenkins or other tool to then apply those changes to either an existing database or create a new database. Our databases are a mixture of on-prem and AWS, with future Azure possible.
I did find a TOAD tool that will be available later this year that seems to do this, however I'm trying to see if there is a less expensive way.
I've been developing applications, databases and supporting the same for 40 years. I can figure it out - however I'd like to not have to recreate the wheel if one exists. Or if there is a best practices, or if there is someone else that has tools they use.
Thanks in advance for any assistance.

The suggested question from over a year ago has some ideas, however it is not close to what I'm looking for. All the answers are over a year old, and as we know software changes at almost a weekly rate. The answers in that question help if I have to "roll my own" solution.

Database migrations are a pretty standard thing for a long time, I believe - not related that much to DevOps, though obviously needed.

There are ready-made tools which are agnostic of your programming environment, i.e., not directly tied to whatever ORM the devs are using.

Strategies of how to actually decide which DDLs/DMLs to feed to the DB might vary; i.e., in the simplest case you just keep track of one global "DB version"; or you have a table which lists individual migrations that have been applied already; or you define a "wanted" state of the DB, and the tool then creates the need DDLs itself to change whatever state the DB is in right now to the new one.

One example of such a standalone tool would be Liquibase (open source), which has a plethora of ways to be configured. I mention that because you say you don't have much time, and it is rather quick to get going - it has one mode where the "configuration" is just a simple file of SQL intermixed with flat (and small) meta information as SQL comments.

The biggest challenge, in my experience is how to handle changing of data structures; one way would be to make all migrations such that they never lose any data, and never break existing code (or data). I.e., you would almost never include a DROP TABLE or ALTER TABLE DROP COLUMN in just any migration, but would let stuff drop out of use, and when a table, or column (or whatever other DB object) finally is not in use anymore, and is inconceivable to be used again, then you drop it. To change, for example, the type of a column, you would create a new one with the correct type, and phase the data over; and so on. This places the work straight on the shoulders of the devs, and not on the migration tool. Which is fine, as this stuff must be in tight correspondence with the DB-centric code of your application, anyways (even if hidden behind an ORM layer).

This is pretty much the same approach as for the software development itself; i.e., creating new features but with a switch (deploying the software with the switch "off", and then being able to toggle quickly), and rolling out parallel implementations of the same feature; only removing the old code when the new one is absolutely proven.


Github commit message when deploying to aws beanstalk in Jenkins As part of our CI setup I am using AWS Beanstalk
deployment plugin to deploy my application through Jenkins.

The issue is I am getting git commit hash in beanstalk
application versions description.
I want to have the commit message instead.
I was able to output the message in Jenkins console output using the execute shell step.
but for some reason I can't seem to transfer it to beanstalk.
If I'm using eb deploy the description contains the commit message.
I am getting a blank description with ${ENV,var="MESSAGE"} in beanstalk.

Would appreciate any help on how to have the commit message
in beanstalk application versions.

This gives a short commit message

git show -s $GIT_COMMIT --format="format:%s"

I'd probably put this in the "Excute shell" command box

Hope this helps!


How do you monitor and alert on thrashing containers? Are there any proposed best practices for how to monitor for thrashing containers? We're in a situation where we have containers that might try to come up and will crash -> restart a few dozen times without anyone noticing. However, it's "normal" for containers to be so ephemeral, so tracking this sort of anomaly seems difficult (what if your workload really is just 30 seconds?)

I'm just wondering if there are any "good enough" alerting practices that anyone else has come up with for tracking unhealthy restarts vs healthy restarts?

Regardless if containers are ephemeral or not, there's two things you could consider not normal:

Non-zero exit codes
Restart count > 0 at the orchestration layer

How you can alert on those metrics will vary wildly between orchestration layers, providers and monitoring tool. I'll keep the discussion about the principles:

Non-zero exit codes

This one seems obvious enough. If you have an ephemeral workload, make sure it exits cleanly and you'll be able to monitor the health of your containers. Even if your ephemeral workload last 30 seconds and you launch thousand of them (small short-lived workers style), as long as everything exits cleanly, there's nothing to worry about.

Restart count at the orchestration layer

Most orchestration layers will expose a restart count. For Kubernetes, that's literally the 'Restart Count' metric when describing a pod. Those restarts are the orchestration layer restarting a container because it wasn't supposed to be in a 'stopped' state, but it was. Ephemeral workloads won't have a restart count, because of their very nature (or if they do, the "final" shutdown of the container won't cause a restart when the workload is done).

Whether you want to alert as soon as the restart count crosses a certain threshold or if you want to be more clever about it and calculate a rate (roughly: restart count / pod/service duration) is up to you.


How to setup multiple build environments in Azure DevOps/TFS2018? So I have an application that contains an ASP.NET Core backend and a React frontend (pure static).

In our TFS 2018, we have created a Build that does the following:

Get sources from the repo
Get Nuget packages (backend)
Build the backend
Run backend tests
Get NPM packages (frontend)
Build the frontend
Run frontend tests
Generate and publish artifacts

Then, a Release pickups those artifacts and deploy them to an Azure AppService (backend) and an Azure Storage v2 account (frontend).

We currently have only one environment, but we need to support multiple ones. The problem is that the frontend build depends on some environment variables which are not present during the build stage. The build doesn't know where is it going to be deployed yet.

So there are a few options here:

Generate the frontend build in a Release. This needs a way for the Release to get access to the sources, either via Git or by getting it from an artifact. Latest is really slow because of all the npm_modules.
Build should produce builds artifacts for all posible environments and each Release environment will pick up the proper one.

What's the recommended approach to handle this on TFS 2018?

This is a general issue with react apps that is independent of what CI/CD stack you use that is discussed at https://stackoverflow.com/questions/49975735/rendering-an-environment-variable-to-the-browser-in-a-react-js-redux-production

With the Azure family of CI/CD products it is expected you create a release artefact then release that same artefact into environments using a release workflow. That is a typical best practice and expected by ”enterprise” release management processes. The general manifesto of native cloud apps is 12factor.net, and it also advises to ”build once, deploy many times, configured by environment variables”. So its a bit of a surprise to me that the best practices isn't really known in the react community so you have to hunt around for solutions.

In my case at that link above I didn't want to make deployments of our react apps different from our backend microservices where we have ”build once, run anywhere”. So we chanced the js code to fetch env vars from the server at page load. Also with this approach and by using containers we are guaranteed to always run the same front end libraries everywhere even if the front end build tool is misconfigured or buggy in terms of not always using the exact same dependency on every run.

In your case you do have a release workflow where you can use all the same built-in and marketplace tasks of a build workflow. This means its straight forward to add a front-end compile task to the release workflows.

On the downside a frontend build of a large react app can take a lot of time and a lot of memory. Memory probably won't be an issue for you. The time taken will be a little annoying. Yet the approach wouldn't be invasive so I would recommend it may be the least effort to set up and the easiest for colleagues to troubleshoot and maintain in the future as long as you have a stable frontend build. Historically npm was less reliable than other frontend builds at giving the exact same lib versions every run for the same package.json (e.g., npm vs. yarn). So you need to study that aspect and ensure that you are certain that every time you run the frontend build it is guaranteed to use the same exact dependency versions.


Best practice for Postgres setup with docker-compose I have a docker-compose script which starts a Postgres container and an app container like so

db:
  image: postgres:10.6
  restart: always
  ports:
    - 5432:5432
app:
  image: test_app:latest
  restart: always
  depends_on:
    - db

This all works fine and 'app' can talk to 'db'. However, the postgres server is in a basically unusable state as it has no configuration with regards to having databases, users, etc. setup (apart from the defaults). It appears that the recommended way of fixing this is to create your own 'db image' pulling from postgres and running some shell script to take care of business (and then your composition uses this image), but this seems very hacky and convoluted to me.

Does anyone have a better way of setting up a postgres server with a known configuration inside a docker-compose based system?

First, use alpine images if possible.

Second, use ENV variables for prepare database configuration. More info.

Third, use /docker-entrypoint-initdb.d directory for extend image. More info.

Example stack with devlopment environment.

version: '3.2'
services:
  example:
    image: "${EXAMPLE_SERVICE}:${EXAMPLE_SERVICE__VERSION}-development"
    build:
      context: .
      dockerfile: Dockerfile.development
    networks:
      - net
    ports:
      - 5000:5000
    depends_on:
      - example__migrator
    stdin_open: true
    tty: true
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: example
      POSTGRES_USER: example
      POSTGRES_HOST: pg.example_net
      POSTGRES_PORT: 5432
      FLASK_APP: app.py
      FLASK_DEBUG: 1
    volumes:
      - .:/home/example
  example__migrator:
    image: "${EXAMPLE_SERVICE}:${EXAMPLE_SERVICE__VERSION}-development"
    build:
      context: .
    command: sh -c './wait-for pg.example_net:5432 -- python manage.py db upgrade'
    networks:
      - net
    depends_on:
      - pg
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: example
      POSTGRES_USER: example
      POSTGRES_HOST: pg.example_net
      POSTGRES_PORT: 5432
  pg:
    image: postgres:10.0-alpine
    networks:
      - net
    ports:
      - 5432:5432
    volumes:
      - pg_data:/var/lib/postgresql/data/pg_data
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: example
      POSTGRES_USER: example
      POSTGRES_HOST: pg.example_net
      POSTGRES_PORT: 5432
      PGDATA: /var/lib/postgresql/data/pg_data
networks:
  net:
volumes:
  pg_data:


Is it possible to set up Microsoft Sql Server in a scalable way? I am working on some architecture where we need to have a Microsoft Sql Server database that is scalable, with one writable master and many read-only slaves.  Ideally I should be able to hit a load balancer of some kind, and if it is a write, it goes to the master, and if it is a read it is load balanced through the slaves.  Some kind of eventual consistency would update the slaves over time to match the data on the master.

I'm fairly sure that AWS and MySQL are capable of doing this out of the box pretty simply.  However, I've got little knowledge of the Microsoft Sql Server side of things.  One of my fellow architects insists it is impossible, but was not able to back up that statement with anything.  We are hosting our microservices in Azure currently, but our databases are supposed to be on prem in our datacenter.  This is a flagship project, so suffice it to say if there's a cool thing out there that we can use for scaling, we can probably convince our DBAs and Ops to use it for this project.

That said, things in the cloud for our DBs are out of the question at the moment.

Not impossible, very easy in fact with read-only routing. Clients intending to be read-only specify it in their connect string, so this is per-session rather than per-statement.  You can have up to 9 replicas in an availability group. With modern hardware and storage, that goes a long way to being “scalable” - 10’s of thousands of concurrent connections easily.


How to pass a String parameter with spaces in Jenkins to environment variable In jenkins Job, I am taking a String parameter as below:

FoldersList = test1,test2,test3,Folder name1,Folder name2,Folder name3

when I pass this value to environment variable the jenkins job fails with error : Unknown lifecycle phase etc etc since spaces are not allowed in folder names.

I am passing the environment variable like this in goals and options section :

exec:java -Dexec.classpathScope="test" -Dexec.mainClass="com.mypackage.main.TestNGRunner" -Dexec.cleanupDaemonThreads=false "-Dexec.args=-inputexcel=Data.xlsx -jenkinsJobName=testJob
 -layer=MultiplefolderTest -InpFoldersList="\"$inpFoldersList"\" -threadCount=12 -retryCount=3"

If I pass a single string within double quotes it works but the user will enter multiple folder names possibly with spaces. Please can somebody help since I tried different options but none work. Is there a way to manipulate the string and pass it to the environment variable.

If the input string is failing due to folders with spaces not having quotes around them, try adding quotes around every object.
The InpFolderList already adds end quotes:
InpFoldersList="\"$inpFoldersList"\"
Add the quotes around every object.
groovy:
class Example {
   static void main(String[] args) {
        String FolderList = "test1,test2,test3,Folder name1,Folder name2,Folder name3";
        println(FolderList.replaceAll(",","\",\""));
   }
}

Output: (End quotes are left off)
test1","test2","test3","Folder name1","Folder name2","Folder name3
Test Code: https://www.jdoodle.com/execute-groovy-online/ (Copy / Paste Code and Execute)
Using above as an example, you could also add the end quotes and have a complete String variable to pass as $InpFoldersList, which would remove confusion if attempting to debug.
Reference: http://docs.groovy-lang.org/latest/html/api/org/codehaus/groovy/runtime/StringGroovyMethods.html#replaceAll(java.lang.CharSequence,java.util.regex.Pattern,java.lang.CharSequence)


Strange orchestration behaviour of a Docker Swarm manager - pile of containers For some unknown reason, the manager host which gets the command to launch a stack, tries to pile up all the containers on itself; other worker/manager hosts stay unemploed.

Unfortunately there are no log entries for that.

Has anybody observed this behaviour already with Docker CE as well?

I would check that all nodes are active and ready with command

docker node ls

With

docker swarm leave

and

docker swarm join \
--token <SWMTKN token> \
<manager private IP>

you may rejoin malfunctioning node.


Whitelist Travis CI to Google Cloud SQL I'm having trouble working out how exactly I can whitelist my TravisCI runner for testing a python adaptor talking to a Google Cloud SQL service. The Cloud SQL will only accept connections from IPs that have been whitelisted ahead of time, though I'm not sure how reserve one from Travis (if such a thing is even possible). How should I go about this?

You can find the IP addresses of the Travis build machines here. If you add the IP addresses of the Travis infrastructure you use to your whitelisted IPs in Cloud SQL it should work. Keep in mind these IP addresses can change in the future though.


How to forcibly remove the resource created by the broken module? I've created few dns_cname_record resources, however Terraform DNS provider started to throw the errors like:

  Error: update server is not set

I've created the bug report for it, as it prevents me from running any Terraform command. The error is basically shown on every Terraform command, including plan.

Now I want to clean up and get rid of this DNS module completely, and all references to it. So I've removed all my dns_cname_record resources from all .tf files. However there are some still resources stored in a state file which I want to get rid of.

$  terraform show | grep -w dns_cname_record
# dns_cname_record.foo-example:
resource "dns_cname_record" "foo-example" {
# dns_cname_record.bar-example:
resource "dns_cname_record" "bar-example" {

So I've tried to destroy it with force without refreshing state, but it failed as well:

$ TF_LOG=trace terraform destroy -target=dns_cname_record.foo-example -force -refresh=false
...
[TRACE] <root>: eval: *terraform.EvalIf
[TRACE] <root>: eval: *terraform.EvalApplyProvisioners
[TRACE] <root>: eval: *terraform.EvalIf
[TRACE] <root>: eval: *terraform.EvalIf
[TRACE] <root>: eval: *terraform.EvalApply
[DEBUG] dns_cname_record.foo-example: applying the planned Delete change
[TRACE] GRPCProvider: ApplyResourceChange
[DEBUG] dns_cname_record.foo-example: apply errored, but we're indicating that via the Error pointer rather than returning it: update server is not set
[TRACE] <root>: eval: *terraform.EvalWriteState
[TRACE] EvalWriteState: writing current state object for dns_cname_record.foo-example
[TRACE] <root>: eval: *terraform.EvalApplyPost
[ERROR] <root>: eval: *terraform.EvalApplyPost, err: update server is not set
[ERROR] <root>: eval: *terraform.EvalSequence, err: update server is not set
[ERROR] <root>: eval: *terraform.EvalOpFilter, err: update server is not set
[TRACE] [walkDestroy] Exiting eval tree: dns_cname_record.foo-example (destroy)
[TRACE] vertex "dns_cname_record.foo-example (destroy)": visit complete
[TRACE] dag/walk: upstream of "dns_cname_record.foo-example (clean up state)" errored, so skipping
[TRACE] dag/walk: upstream of "provider.dns (close)" errored, so skipping
[TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
[TRACE] dag/walk: upstream of "root" errored, so skipping

Error: update server is not set

Releasing state lock. This may take a few moments...
[DEBUG] plugin: plugin process exited: path=~/.terraform/plugins/darwin_amd64/terraform-provider-dns_v2.1.1_x4 pid=76229
[DEBUG] plugin: plugin exited

and at the end, it doesn't remove invalid entry, because of the error.

I've also tried to remove terraform-provider-dns_v2.1.1_x4 binary module file, but teraform plan complains about it, and terraform init downloads it again, and the problem repeats.

Is there any higher power command than destroy -force -refresh=false to get rid of the invalid entries from the state file which breaking local setup? Or I need to edit the state file manually?

The workaround is to pull the current state file, edit it and use it as a base line. For example:

$ terraform state pull > terraform.tfstate
$ vim terraform.tfstate # Carefully remove invalid entries.
$ python -mjson.tool terraform.tfstate # Validate JSON.

Note: In Vim, placing cursor on the opening bracket, hitting d% will remove the whole group.

Then plan and apply:

$ terraform plan -state=terraform.tfstate -refresh=false
$ terraform apply -state=terraform.tfstate -refresh=false

Alternatively, pull, edit and push your local state file into remote state:

$ terraform state pull > terraform.tfstate
$ vim terraform.tfstate # Remove invalid entries and increase the serial value.
$ terraform state push terraform.tfstate

For Azure, it's better to use terraform-provider-azurerm instead of general DNS provider (terraform-provider-dns).