Returning a list of the values from the binary tree I want to return a list of the values from the binary tree.  Is there a shorter and more efficient way to write the method for numbers?

class BTNode(object):
    """A node in a binary tree."""

    def __init__(self, item, left=None, right=None):
        """(BTNode, object, BTNode, BTNode) -> NoneType
        Initialize this node to store item and have children left and right,
        as well as depth 0.
        """
        self.item = item
        self.left = left
        self.right = right
        self.depth = 0  # the depth of this node in a tree

    def number(self: 'BTNode') -> list:
        lst = []

        if self.right is None and self.left is None:
            lst.append(self.item)
        else:
            lst.append(self.item)
        if self.left:
            left = self.left.number()
            lst.extend(left)
        if self.right:
            right = self.right.number()
            lst.extend(right)
        return lst

There is no docstring for the number method.
There are no test cases. This kind of code would be an ideal opportunity to write some doctests.
The depth property is always 0. This seems useless.
The method name number is misleading. (It does not return a number.) I would call it something like flattened_pre_order because it flattens the tree into a list using pre-order traversal.
The variable name lst is misspelt. Better to call it something like result.
This code:

if self.right is None and self.left is None:
    lst.append(self.item)
else:
    lst.append(self.item)

can be replaced with:

lst.append(self.item)

since it doesn't matter whether the condition is true or false.
There's quite a lot of copying going on in the traversal of the tree: each time you call a.extend(b), Python has to copy out the list b. This copying causes the traversal of a tree with n nodes to take O(n2) time instead of O(n). See below for a way to avoid this copying.
You use recursion to visit the child nodes, which means that traversing a very deep tree will exceed Python's maximum recursion depth:

>>> n = None
>>> for i in range(1000): n = BTNode(i, n)
...
>>> n.number()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cr33662.py", line 22, in number
    left = self.left.number()
  [... many lines omitted ...]
RuntimeError: maximum recursion depth exceeded

The way to avoid this is to maintain your own stack, like this:

def flattened_pre_order(self: 'BTNode') -> list:
    """Return a list of items in the tree rooted at this node, using
    pre-order traversal.

        >>> tree = BTNode(1, BTNode(2, None, BTNode(3)), BTNode(4))
        >>> tree.flattened_pre_order()
        [1, 2, 3, 4]
        >>> node = BTNode(9999)
        >>> for i in range(9998, -1, -1): node = BTNode(i, node)
        >>> node.flattened_pre_order() == list(range(10000))
        True

    """
    result = []
    stack = [self]
    node = None
    while stack:
        if node is None:
            node = stack.pop()
        if node is not None:
            result.append(node.item)
            stack.append(node.right)
            node = node.left
    return result


Find relative filenames recursively in Emacs I have a set of directories. For each directory I have a set of relative file names. I would like to select any of these relative filenames using completion, and then later reconstruct the absolute path of the filename.

In order to create a list of relative filenames for a given directory, I created this function:

(defun my-get-dir-files (dir pat &optional rel-dir)
"Finds relative filenames. Returns a list of relative filenames/pathnames to
 directory `dir' matching pattern `pat'. The variable `rel-dir` is only used internally
 by recursive calls and should usually not be supplied.

IMPORTANT: This function uses recursion. To avoid excessive copying of lists,
 the variable `files' is used as a dynamic list variable, and is assumed to be
 defined as non-lexical dynamic variable in the caller using a let binding"
  (let ((dir-files nil)
        (cur-dir nil))
    (unless rel-dir (setq rel-dir ""))
    (setq cur-dir (expand-file-name rel-dir dir))
    (setq dir-files (directory-files cur-dir nil nil t))
    (dolist (name dir-files)
      (unless (or (string= name ".") (string= name ".."))
        (let ((fn (expand-file-name name cur-dir)))
          (if (file-directory-p fn)
              (let ((new-rel-dir nil))
                (if (string= "" rel-dir)
                    (setq new-rel-dir name)
                  (setq new-rel-dir (concat rel-dir "/" name)))
                (my-get-dir-files dir pat new-rel-dir)) ;; recursive call
            (when (string-match pat name)
              (let ((rel-fn nil))
                (if (string= "" rel-dir)
                    (setq rel-fn name)
                  (setq rel-fn (concat rel-dir "/" name))
              (setq files (append files `(,rel-fn))))))))))))

Then I plan to use code like:

(let ((dir "/home/hakon/dir1")
      (files nil))
  (my-get-dir-files dir "\\.el$")
  files)

to set the list variable files.

One thing that worries me here is the use of the (non-lexical) dynamic variable files. Can or should this be avoided?

According to the manual I have violated this principle:

  If a variable has no global definition, use it as a local variable
  only within a binding construct

Yeah don't use a dynamic variable for that, it's just confusing for the
user.

It is way easier to deal with input and output from a
function instead of having to take special variables into account, basically it's now part of your interface.  And I personally
would be really angry if a random function somewhere took a common
variable like files from outside its scope and filled it with data.

So, after that explanation, I would recommend using either a separate
function, or cl-flet/cl-labels to handle the recursive part and only
expose the necessary parts to the caller.  That means no optional
rel-dir, which isn't really part of the interface and no using files
from the outer scope.

To wit:

(defun my-get-dir-files (dir pat)
  "..."
  (let (files)
    (cl-flet ((aux (dir pat rel-dir)
                    (let ((dir-files nil)
                          (cur-dir nil))
                      (unless rel-dir (setq rel-dir ""))
                      (setq cur-dir (expand-file-name rel-dir dir))
                      (setq dir-files (directory-files cur-dir nil nil t))
                      (dolist (name dir-files)
                        (unless (or (string= name ".") (string= name ".."))
                          (let ((fn (expand-file-name name cur-dir)))
                            (if (file-directory-p fn)
                                (let ((new-rel-dir nil))
                                  (if (string= "" rel-dir)
                                      (setq new-rel-dir name)
                                    (setq new-rel-dir (concat rel-dir "/" name)))
                                  (aux dir pat new-rel-dir)) ;; recursive call
                              (when (string-match pat name)
                                (let ((rel-fn nil))
                                  (if (string= "" rel-dir)
                                      (setq rel-fn name)
                                    (setq rel-fn (concat rel-dir "/" name))
                                    (setq files (append files `(,rel-fn)))))))))))))
      (aux dir pat nil)
      files)))

Note that I just wrapped the existing function one time and put a let
around it to keep the side effect more contained.  And now that files
is lexical, everything is fine.

But really, it would be better to just use an accumulator.  You
only ever add one element to the list, in which case you should just
immediately use push instead (and if you care about the output order,
nreverse the final result).

The following changes could also be made later:

Use member instead of the or, less typing for you.
(Short variable names don't necessarily improve readability.  Even
though Elisp code seems to disagree in places.)
The explicit binding to nil isn't necessary as it's the default
anyway.
Also the setqs there are unnecessary, just put the initial binding
into the let instead, use or and other functional tools.
cond is more readable for nested conditions, if applicable.
I would use list for simple list constructions like (,foo).
(setq rel-fn name) in the if doesn't do anything as it's the only form in the true branch.

So with all that it would be like this (even though personally I'd
rename the fn to filename as well.  YMMV:

(defun my-get-dir-files (dir pat)
  (let (files)
    (cl-flet ((aux (dir path rel-dir)
                   (let ((cur-dir (expand-file-name rel-dir dir)))
                     (dolist (name (directory-files cur-dir nil nil t))
                       (unless (member name '("." ".."))
                         (let ((fn (expand-file-name name cur-dir)))
                           (cond
                            ((file-directory-p fn)
                             (aux dir pat (if (string= "" rel-dir)
                                              name
                                            (concat rel-dir "/" name))))
                            ((string-match pat name)
                             (unless (string= "" rel-dir)
                               (push (concat rel-dir "/" name) files))))))))))
      (aux dir pat "")
      files)))

I'm pretty sure all transformations kept the meaning the same; a quick
check via

(set-exclusive-or (let (files) (my-get-dir-files-1 ".." ".*") files) (my-get-dir-files ".." ".*") :test #'equal)

with -1 being the original function resulted in no difference in the
lists (except for order of course), so that was enough for me.  If you
return (nreverse files) instead the lists are exactly equal.

And at this point I'm done.  There might still be potential
optimisations during the concatenations and so on.  The recursion itself here is fine as you won't encounter directory trees deep enough to cause the recursion to fail and it's also fairly efficient now, so no need to change anything there.

You could still extract a local function for (if (string= "" rel-dir) ...) and use that there and in the other branch of the cond.

Next up:  Would be cool if the function also handled "." for dir.


Python web crawler counting words in all pages in a domain Here is a simple web crawler I wrote in Python 3 that counts the words in each page in a domain, displays word count per page, and sums them up.

Tested on a multitude of web pages, works correctly so far.

What do you think of my code?

import requests
from urllib.request import urlopen
from bs4 import BeautifulSoup

base_url = "https://independencerpgs.com/"
html = urlopen(base_url)
bsObj = BeautifulSoup(html.read(), features="html.parser");

urls=[]
for link in bsObj.find_all('a'):
    if link.get('href') not in urls:
        urls.append(link.get('href'))
    else:
        pass
print(urls)

words=0
for url in urls:
    if url not in ["NULL", "_blank", "None", None, "NoneType"]:
        if url[0] == "/":
            url=url[1:]
        if base_url in url:
            if base_url == url:
                pass
            if base_url != url and "https://"in url:
                url=url[len(base_url)-1:]
        if "http://" in url:
            specific_url=url
        elif "https://" in url:
            specific_url = url
        else:
            specific_url = base_url + url
        r = requests.get(specific_url)
        soup = BeautifulSoup(r.text, features="html.parser")
        for script in soup(["script", "style"]):
            script.extract()
        text = soup.get_text()
        lines = (line.strip() for line in text.splitlines())
        chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
        text = '\n'.join(chunk for chunk in chunks if chunk)
        text_list = text.split()
        print(f"{specific_url}: {len(text_list)} words")
        words += len(text_list)
    else:
        pass
print(f"Total: {words} words")

It's readable and gets the job done as a script, but there is some redundancy in your if cases. Here's a streamlined version:

import requests
from urllib.request import urlopen
from bs4 import BeautifulSoup

base_url = "https://independencerpgs.com/"
html = urlopen(base_url)
bsObj = BeautifulSoup(html.read(), features="html.parser");

# Sets handle duplicate items automatically
urls=set()
for link in bsObj.find_all('a'):
    urls.add(link.get('href'))
print(urls)

words=0
for url in urls:
    # Having the logic this way saves 4 spaces indent on all the logic
    if url in ["NULL", "_blank", "None", None, "NoneType", base_url]:
        continue

    # The == base_url case is handled in the above `if`
    if url[0] == "/":
        specific_url = base_url + url # requests.get does not care about the num of '/'
    else:
        specific_url = url

    r = requests.get(specific_url)
    soup = BeautifulSoup(r.text, features="html.parser")
    for script in soup(["script", "style"]):
        # Use clear rather than extract
        script.clear()
    # text is text you don't need to preprocess it just yet.
    text = soup.get_text()
    print(f"{specific_url}: {len(text)} words")
    words += len(text)

print(f"Total: {words} words")

Depending on what you intend to do with that code, you might want to put it into functions or even create a class around it too.


Cross Thread Access to Object in ViewModel This is a fairly straight forward question. I have a C# WPF MVVM application that call a C++ DLL to do some compilation/crunching. To update the UI C# application for progress updates I am using callback delegates.

public static class Callbacks
{
    public static ConsoleCallback ConsoleOutputCallback { get; set; }

    [UnmanagedFunctionPointer(CallingConvention.StdCall)]
    public delegate void ConsoleCallback(string message);
}

where the implementation is

Callbacks.ConsoleOutputCallback = (message) =>
    {
        console.Document.Text += message;
    };

where Document is of type TextDocument. I am then passing this object into the C++ to be able to invoke it and update the UI. This works great, however, for one control I am getting a standard

  A first chance exception of type 'System.InvalidOperationException' ... TextDocument can be accessed only from the thread that owns it.

Now, I understand this error. To get around it I have done the following

TaskScheduler scheduler = TaskScheduler.FromCurrentSynchronizationContext();
Callbacks.ConsoleOutputCallback = (message) =>
    {
        Task.Factory.StartNew(() =>
        {
            console.Document.Text += message;
        }, CancellationToken.None,
           TaskCreationOptions.None,
           scheduler);
    };

which works. However, it would be nicer to look at if I could do a similar thing that I could do with a Control (which TextDocument Document is not) like

public static void PerformSafely<T>(this Control target, Action<T> action, T parameter)
{
    if (target.InvokeRequired)
        target.Invoke(action, parameter);
    else
        action(parameter);
}

but I cannot do this with TextDocument as it does not inherit from Control. Is there a nicer way to do what I am doing?

The TPL is unfortunately verbose at times. The only way I can think of to make it look "nicer" is to point ConsoleOutputCallback to a named method and delegate the creation of the Task to another method:

public static void MyMethod()
{
    Callbacks.ConsoleOutputCallback = ConsoleOutputCallback;
}

public static void ConsoleOutputCallback(string message)
{
    var scheduler = TaskScheduler.FromCurrentSynchronizationContext();

    Dispatch(
        msg => console.Document.Text += msg,
        message,
        scheduler
    );

    Dispatch(
        msg => someOtherNonControl.Text += msg,
        message,
        scheduler
    );
}

public static void Dispatch<T>(Action<T> action, T parameter, TaskScheduler scheduler)
{
    Task.Factory.StartNew(
        () => action(parameter),
        CancellationToken.None,
        TaskCreationOptions.None,
        scheduler
    );
}


Tabbed navigation to hide and show pages I'm new to jQuery and trying to learn how to refactor my bloated code to make it nicer and better maintainable.

I have a tabbed navigation which I'm using jQuery to hide and show pages depending on the selector that is clicked.

  $('#search_distributors').on 'click', (e) =>
    e.preventDefault()
    $('#distributor_location_search_filter_form').show()
    $('#distributor_location_search_container').show()
    $('#international_distributors_location_container').hide()
    $('#all_distributors_location_container').hide()
    $('li.search_locations').addClass('active')
    $('li.international').removeClass('active')
    $('li.all_distributors').removeClass('active')

  $('#international_distributors').on 'click', (e) =>
    e.preventDefault()
    $('#international_distributors_location_container').show()
    $('#distributor_location_search_container,
      #distributor_location_search_filter_form,
      #all_distributors_location_container').hide()
    $('li.search_locations').removeClass('active')
    $('li.all_distributors').removeClass('active')
    $('li.international').addClass('active')

    $('#all_distributors').on 'click', (e) =>
      e.preventDefault()
      $('#all_distributors_location_container').show()
      $('#international_distributors_location_container,
        #distributor_location_search_container,
        #distributor_location_search_filter_form, ').hide()
      $('li.search_locations').removeClass('active')
      $('li.international').removeClass('active')
      $('li.all_distributors').addClass('active')

I'm familiar with using functions and stuff but unsure how to DRY this up.  If anyone can shed some light on this and help me learn that would be wonderful.

It's hard to give an exact answer without knowing your HTML structure, but if we assume that:

Your tab headers (apparently li-elements with classes like .search_locations and ids like #search_distributors) have some common class, such as .tabheader.
Each of your tabs is wrapped in an element with a common class, such as .tab.
You can add a HTML5 data- attribute on your tab headers to designate the tab whiches header it is.

Assumptions 1. and 2. are necessary so that you can use a common function for all the tabs without writing what is essentially the same code thrice.

Number 3. could be implemented like this:

Search locations

Then you can reduce your CoffeeScript to

$('.tabheader').on 'click', (e) =>
    e.preventDefault()
    tabId = $(e.currentTarget).data('tabId')
    $('.tab').hide()
    $('#' + tabId).show()
    $('.tabheader').removeClass('active')
    $(e.currentTarget).addClass('active')

See my jsFiddle.

If you, some strange reason, would not be able to alter your HTML in any way, your code would still be easier to maintain if you wrote a common function for displaying a tab, then called it for each of the click events:

showTab = (e, tab, header) ->
    e.preventDefault()
    allTabs = [
        '#distributor_location_search_filter_form',
        '#distributor_location_search_container',
        '#international_distributors_location_container',
        '#all_distributors_location_container'
    ]
    $(allTabs.join(',')).hide()
    if tab instanceof Array
        tab = tab.join(',')
    $(tab).show()
    allHeaders = [
        'li.search_locations',
        'li.international',
        'li.all_distributors'
    ]
    $(allHeaders.join(',')).removeClass('active')
    $(header).addClass('active')

$('#search_distributors').on 'click', (e) =>
    showTab(e, ['#distributor_location_search_filter_form', '#distributor_location_search_container'], 'li.search_locations')

$('#international_distributors').on 'click', (e) =>
    showTab(e, '#international_distributors_location_container', 'li.international')

$('#all_distributors').on 'click', (e) =>
    showTab(e, '#all_distributors_location_container', 'li.all_distributors')

See my other jsFiddle.

Note that there are also multiple jQuery plugins for tab behavior, such as one in Bootstrap and jQuery UI.


Inserting and updating data for a website monitoring app I have created an app in which URLs stored in a database are pinged via curl to check their availability and the database is updated with the results.

Two tables are updated with each fresh ping of a website (Site -- the status recieved 200, 404 etc) and (History - time of each ping, time taken, result etc) From my understanding because I am using Doctrine I have to create an object each time and update the database one object at a time. Is there another way around this? This solution means for each site I have I must create two queries for each and this could quickly get out of hand.

About the Database

Site and History have a many to one relationship.  One Site can have many history entries but each history only has one site relating to it.

Is there a more efficient way of doing this?

Default Controller.php

<?php

namespace AppBundle\Controller;

use AppBundle\Controller\Base\Controller;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;

use AppBundle\Entity\Site;
 use AppBundle\Entity\History;
use AppBundle\Entity\Status;

class DefaultController extends Controller
{

/**
 * @Route("/scan", name="homepage")
 */
public function indexAction()
{
    $allSites = $this->getSites(); // Returns sites repo
    $allStatuses = $this->getStatuses(); // returns all possible statusus 404, 200 etc

    $results = $this->getScanManager()->scanBatch($allSites); // scans all the sites and returns results array

    foreach ($results as $result) {

        $history = new History();
        $history->setSiteId($this->findSiteById($result['id'], $allSites)); // RETURNS THE CORRESPONDING SITE OBJECT - SINCE IT IS AN ID
        $history->setTime(date('h:i:s d-m-Y'));
        $history->setStatusId($this->findStatusById($result['status'], $allStatuses));//RETURNS THE CORRESPONDING STATUS OBJECT - SINCE IT IS AN ID
        $history->setResponseTime($result['speed_download']);

        $em = $this->getDoctrine()->getManager();
        $em->persist($history);
        $em->flush();

        $site = $this->findSiteById($result['id'], $allSites);
        $site->setSiteStatus($this->findStatusById($result['status'], $allStatuses));
        $site->setSitePingTime($result['speed_download']);
        $site->setSiteLastUpdate(date('h:i:s d-m-Y'));

        $em = $this->getDoctrine()->getManager();
        $em->persist($site);
        $em->flush();

    }

    return $this->render('default/index.html.twig');
}

}

You can optimize it so that it's just 2 DB queries:

    $allSites = $this->get('SiteRepository')->CustomQuery(); // Returns Site array indexed by id

    $allStatuses = $this->get('StatusRepository')->CustomQuery(); // Returns Status array indexed by id

    $results = $this->getScanManager()->scanBatch($allSites); // scans all the sites and returns results array

    $em = $this->getDoctrine()->getManager();

    foreach ($results as $result) {

        $history = new History();
        $history->setSite($allSites[$result['id']]);
        $history->setTime(date('h:i:s d-m-Y'));
        $history->setStatus($allStatuses[$result['status']]);
        $history->setResponseTime($result['speed_download']);

        $em->persist($history);

        $site = $this->findSiteById($result['id'], $allSites);
        $site->setSiteStatus($allStatuses[$result['status']]);
        $site->setSitePingTime($result['speed_download']);
        $site->setSiteLastUpdate(date('h:i:s d-m-Y'));
    }

    $em->flush();

    return $this->render('default/index.html.twig');

Of course this should be adjusted to your app, if you allways ping all the sites, then it's ok, but i you just ping a few, then it might be inefficient, but at least you get the idea.

BTW, if your setter methods return $this, you could write it in a prettier way:

$history->setSite($allSites[$result['id']])
   ->setTime(date('h:i:s d-m-Y'))
   ->setStatus($allStatuses[$result['status']])
   ->setResponseTime($result['speed_download']);


Subsorting with multiple Comparators I've seen a number of questions on Stack Overflow of the kind of "How do I sort on fields X AND Y?" The most common answers to those questions consists of "Write a custom Comparator."

But what if you have 3 fields you can sort on? Or 13? or 300? My solution is this utility class, so that rather than writing a billion different Comparators for every possible combination you might want, you write just 3 (or 13, etc.) simple ones and essentially composite them to get the desired behavior.

This method is designed to be used as essentially an expanded version of Collections.sort(List list, Comparator c) so I want to adhere to the same contract as much as possible.

I am also interested in ways to improve the documentation/API and finding possibly troublesome corner cases.

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;

/**
 * A utility class for sorting a list using multiple sorting criteria.
 *
 */
public class Multisorter {
    /**
     * Sorts the given List using the given Comparators.
     *
     * The List is sorted by the Comparators in the order the Comparators are
     * given; the elements are sorted according to the first Comparator, then
     * all of the elements that are considered "equal" are subsorted recursively
     * by the next Comparator and so on.
     *
     * This method makes use of Collections.sort(List<T>, Comparator<T>)
     * internally and obeys the same contract.
     *
     * @param <T>
     *            the generic type of the list.
     *
     * @param list
     *            the List to sort
     * @param comparators
     *            the comparators to sort with
     * @return the sorted List
     */
    public static <T> void sort(List<T> list, Comparator<? super T>... comparators) {
        if (comparators.length <= 0) {
            throw new IllegalArgumentException("Must provide at least one Comparator to sort with.");
        }
        Collections.sort(list, comparators[0]);
        if (comparators.length > 1) {
            List<List<T>> subCollections = new ArrayList<>();
            for (T element : list) {
                boolean matchFound = false;
                for (List<T> subList : subCollections) {
                    if (comparators[0].compare(element, subList.get(0)) == 0) {
                        subList.add(element);
                        matchFound = true;
                    }
                    if (matchFound) {
                        break;
                    }
                }
                if (!matchFound) {
                    int lastIndex = subCollections.size();
                    subCollections.add(lastIndex, new ArrayList<>());
                    subCollections.get(lastIndex).add(element);
                }
            }

            // Sort each of the subcollections recursively without the first Comparator
            for (List<T> subList : subCollections) {
                sort(subList, Arrays.copyOfRange(comparators, 1, comparators.length));
            }
            // Aggregate all the subcollections
            list.clear();
            for (List<T> subList : subCollections) {
                list.addAll(subList);
            }
        }
    }
}

However, based on comments by Alnitak and rolfl, I would just use the new thenComparing method in Comparator added in Java 8 that I didn't know about.

I find your code fairly complicated. Especially the part where you split the list into sequences of sublists of equal elements is a bit convoluted. Here's what you're doing in pseudocode:

given a comparator,
for each element in the list:
    for each sublist:
       if per the comparator, the element is equal to an element in that sublist:
           add the element to the sublist,
           continue with next element
    if no sublist matched:
        add a new sublist containing that element

The most straightforward way to implement that in Java is using a labeled loop. Using that, we can continue with the next element in the outer loop once a match was found. We do not have to maintain a matchFound variable:

List<List<T>> subCollections = new ArrayList<>();

element:
for (T element : list) {
    for (List<T> subList : subCollections) {
        if (comparators[0].compare(element, subList.get(0)) == 0) {
            subList.add(element);
            continue element;
        }
    }
    // no sublist matched
    List<T> newList = new ArrayList<>();
    newList.add(element);
    subCollections.add(newList);
}

There's also no need to use a lastIndex here.

Unfortunately, what you are doing is tremendously inefficient. You re-sort each list multiple times, and invoke each comparator unnecessarily often. You are more or less implementing a crazy inefficient multi-level quicksort that partitions into multiple parts per level.

The better solution would be to create a custom comparator that loops through the list of comparators and stops when they are no longer equal:

class MultiComparator<T> implements Comparator<T> {
    private final List<Comparator<T>> comparators;

    public MultiComparator(List<Comparator<? super T>> comparators) {
        this.comparators = comparators;
    }

    public MultiComparator(Comparator<? super T>... comparators) {
        this(Arrays.asList(comparators));
    }

    public int compare(T o1, T o2) {
        for (Comparator<T> c : comparators) {
            int result = c.compare(o1, o2);
            if (result != 0) {
                return result;
            }
        }
        return 0;
    }

    public static <T> void sort(List<T> list, Comparator<? super T>... comparators) {
        Collections.sort(list, new MultiComparator<T>(comparators));
    }
}

(untested)


Counting only weekend days between two timestamps I am fairly new to the Groovy arena. I recently modified some code to add the following block. This code is added to remove dates we were supposed to remove from the total counts.  This data goes on reports that a bug was noticed, that we weren't removing the queue entry dates.

result.processed.each{
    def queueEntry = QueueEntry.findById(it.id)<<<START ADD>>>
    Set dates = new HashSet<Long>()

    def children = QueueEntry.findAllByParent(queueEntry)

    for(QueueEntry qe : children){
        def f = new GregorianCalendar()
        f.setTimeInMillis(DateUtils.getClearedTime(qe.entryTimestamp))
        def l = new GregorianCalendar()
        l.setTimeInMillis(DateUtils.getClearedTime(qe.exitTimestamp))
        while(f < l){
            if(f.get(Calendar.DAY_OF_WEEK) != Calendar.SUNDAY && f.get(Calendar.DAY_OF_WEEK) != Calendar.SATURDAY){//only add weekdays
                dates.add(f.time.time)
            }
            def xx = new GregorianCalendar()
            xx.setTimeInMillis(f.time.next().time)
            f = xx
        }
        dates.add(l.time.time)
    } <<<STOP ADD>>>
    Set outsideDays = it.numberOfDaysOutsideCVB
    Set days = DateUtils.businessDaysBetweenDates(it.entryTimestamp, it.exitTimestamp)
    days.removeAll(outsideDays)
    days.removeAll(dates)
    turnTimes << days.size()
}

The application is now crawling. I am obviously doing something wrong. When this is run for small data-sets it will complete slowly. On larger sets it doesn't finish. Prior to this change it was completing.

I found the logic of the set operations a bit hard confusing.  It appears that the goal is to take some set of days, named days, and remove entries from it.  So, start with days and remove entries from it.  You don't need the other sets named outsideDays and dates.

It's weird that it.numberOfDaysOutsideCVB returns a set rather than a number, as suggested by the method name.

The looping mechanism is cumbersome:

def xx = new GregorianCalendar()
xx.setTimeInMillis(f.time.next().time)
f = xx

The l day is always removed, regardless of the day of the week.  That deserves an explanatory comment.  Furthermore, f and l are somewhat cryptic variable names.

Set days = DateUtils.businessDaysBetweenDates(it.entryTimestamp, it.exitTimestamp)
days.removeAll(it.numberOfDaysOutsideCVB)

def children = QueueEntry.findAllByParent(QueueEntry.findById(it.id))
for (QueueEntry qe : children) {
    def entry = new GregorianCalendar()
    entry.setTimeInMillis(DateUtils.getClearedTime(qe.entryTimestamp))
    def exit = new GregorianCalendar()
    exit.setTimeInMillis(DateUtils.getClearedTime(qe.exitTimestamp))

    entry.upto(exit.previous()) { day ->
        switch (day.get(Calendar.DAY_OF_WEEK)) {
          case Calendar.SATURDAY:
          case Calendar.SUNDAY:
            break
          default:
            days.remove(day.time.time)
        }
    }
    // Remove the last day regardless of day of week. (Why?)
    days.remove(exit.time.time)
}
turnTimes << days.size()


Select Records Constituting Percentage of Total The Setup
I have a table (call it myTable) in MS Access with two fields: Item (Text) and Quantity (Long).
For example, the first 10 records might be:
Item      |  Quantity
----------+----------
Item001   |       706
Item002   |       533
Item003   |       580
Item004   |       289
Item005   |       302
Item006   |       775
Item007   |        14
Item008   |       761
Item009   |       815
Item010   |       709

The Task
I'm looking to obtain a table/query containing the top N records whose Quantity constitutes a given percentage of the total Quantity across all records.
For example:
If the total of the Quantity field for all records is 1000 and the given percentage is 80%, the output table/query would contain sufficient records in decreasing Quantity until the total Quantity is 800.

My Code
Here is my current approach:
Function Percentile(dblPct As Double)
    On Error GoTo Cleanup
    Dim dbsCdb As DAO.Database
    Dim rstRs1 As DAO.Recordset
    Dim rstRs2 As DAO.Recordset
    Dim lngTot As Long
    Dim lngSum As Long
    Dim lngRcd As Long

    Set dbsCdb = CurrentDb
    Set rstRs1 = dbsCdb.OpenRecordset("SELECT Sum(Quantity) FROM myTable")

    With rstRs1
        If Not .EOF Then
            .MoveFirst
            lngTot = .Fields(0) * dblPct
        End If
        .Close
    End With
    Set rstRs1 = Nothing

    If 0 < lngTot Then
        Set rstRs2 = dbsCdb.OpenRecordset("SELECT * FROM myTable ORDER BY Quantity DESC")
        With rstRs2
            If Not .EOF Then
                .MoveFirst
                Do Until .EOF Or lngSum >= lngTot
                    lngRcd = lngRcd + 1
                    lngSum = lngSum + !Quantity
                    .MoveNext
                Loop
            End If
            .Close
        End With
        Set rstRs2 = Nothing

        DoCmd.RunSQL "SELECT TOP " & lngRcd & " * INTO OutputTable FROM myTable ORDER BY Quantity DESC"
    End If

Cleanup:
    On Error Resume Next
    If Not (rstRs1 Is Nothing) Then rstRs1.Close: Set rstRs1 = Nothing
    If Not (rstRs2 Is Nothing) Then rstRs2.Close: Set rstRs2 = Nothing
    If Not (dbsCdb Is Nothing) Then Set dbsCdb = Nothing
End Function

I first calculate the total Quantity of all records in myTable and multiply this by the given percentage argument (e.g. 0.8 for 80%).
I then iterate over the records in descending order of Quantity and calculate a running total, ceasing when this total is greater than or equal to the target, counting the number of records it takes to do so.
Finally, I select the calculated number of records from the table to create an OutputTable.

What I'm Hoping For

I'm relatively new to VBA, so feedback on any obvious bad practices in my code would be appreciated.

I feel like I'm missing something obvious - could this be achieved using one or two SQL statements?

Many thanks in advance for your time!

This is very good question and a very hard one because your query is basically sensitive to the ordering. In other SQL database engines, this is easier to solve with use of window functions. Unfortunately, Access database engine does not support window functions.

A possible choice is to use subquery but this can get very expensive quickly because a subquery would have to be correlated and therefore re-calculated for all possible rows, making it O(too much n). For completeness' sake, here's a possible query:

SELECT
  m.Item,
  m.Quantity,
  (
    SELECT SUM(x.Quantity)
    FROM myTable AS x
    WHERE x.Quantity >= m.Quantity
  ) AS RunningSum,
  (
    SELECT SUM(t.Quantity)
    FROM myTable AS t
  ) * 0.8 AS PercentileThreshold
FROM myTable AS m
WHERE (
  SELECT SUM(x.Quantity)
  FROM myTable AS x
  WHERE x.Quantity >= m.Quantity
) <= (
  SELECT SUM(t.Quantity)
  FROM myTable AS t
) * 0.8
ORDER BY m.Quantity DESC;

To avoid paying the expenses of multiple queries being executed several times over, you can combine some VBA with SQL. For instance, you can create the following VBA in a public standard module:

Private Type TPercentile
    PercentileThreshold As Currency
    PercentileValues As Scripting.Dictionary
End Type
Private This As TPercentile

Public Function StartPercentile(PercentileThreshold As Currency) As Boolean
    This.PercentileThreshold = Nz(DSum("Quantity", "myTable") * PercentileThreshold, 0)
    Set This.PercentileValues = New Scripting.Dictionary
    StartPercentile = True
End Function

Public Function Percentile(Key As String, Quantity As Variant) As Currency
    Dim PercentileValue As Currency

    If Not This.PercentileValues.Exists(Key) Then
        PercentileValue = Nz(DSum("Quantity", "myTable", "Quantity>=" & Nz(Quantity, 0)))
        This.PercentileValues.Add Key, PercentileValue
    End If
    Percentile = This.PercentileValues(Key)
End Function

Public Function PercentileThreshold() As Currency
    PercentileThreshold = This.PercentileThreshold
End Function

Your query then becomes:

SELECT
  m.Item,
  m.Quantity,
  Percentile(m.Item,m.Quantity) AS Percentile,
  PercentileThreshold() AS Threshold
FROM myTable AS m
WHERE StartPercentile(0.8)
  AND Percentile([m].[Item],[m].[Quantity]) <= PercentileThreshold()
ORDER BY m.Quantity DESC;

This enables you to handle a larger dataset by populating the dictionary only once and require less code to boot and you can easily generalize for other tables. The downside is that this is limited to only one active running query at a time. If this is unacceptable, then you need to add some kind of identifier that each query can then generate in their invocation to StartPercentile() function so that they have their own "instances".

I also want to call your attention to one possible option for you - If you are reporting, Access already provides you with a Running Sum property on the Report so you could conceivably create a simple query, then apply a filter on a calculated control, rather than doing in the query. That yields much less code.

You may wonder why I use Currency all over the place -- the answer is that I generally avoid using floating numbers such as Double which can cause unstable results due to rounding errors. Currency is a scaled integer and thus is not subject to such problems so it is a great data type to use for where you need fractions less than 4 decimal places. I should note that there is also a Decimal type but IME, it yields more problems than solutions and I never use it within Access/VBA context.


Implementing Addition, subtraction, multiplication, division of numbers of different base and give result in desired base: Implement addition, subtraction, division and multiplication of given numbers of different base and provide result in desired base:

Example:
n1,b1 = 1010, 2
n2,b2 = 100, 10
desired_base = 9
addition(n1,b1,n2,b2,desired_base) = 22
subtraction(n1,b1,n2,b2,desired_base) = 0
multiplication(n1,b1,n2,b2,desired_base) = 121

I have written this code, but How to improve its complexity?
global dict_base, dict_reverse
dict_base = {'0': 0, '1':1, '2':2, '3':3, '4':4, '5':5, '6':6,'7':7,'8':8, '9':9, 'A':10, 'B':11, 'C':12, 'D':13, 'E':14, 'F':15, 'G':16, 'H':17}
dict_reverse = {v: k for k, v in dict_base.iteritems()}
def convert_to_base_10(num_str, base):
    global dict_base
    k = 1
    ans = 0
    for char in num_str[::-1]:
        ans += dict_base[char] * k
        k = k * base
    return int(ans)

def convert_from_base_10(num, des_base):
    global dict_reverse
    ans = []
    neg_flag = False
    import pdb; pdb.set_trace()
    if num < 0:
        neg_flag = True
        num = num * (-1)
    while num>0:
        ans.append(dict_reverse[num % des_base])
        num = num / des_base
    if neg_flag:
        return '-'+''.join(str(digit) for digit in ans[::-1])
    return ''.join(str(digit) for digit in ans[::-1])

def addition(n1, b1, n2, b2, b):
    n1_10 = convert_to_base_10(n1,b1)
    n2_10 = convert_to_base_10(n1,b1)
    ans = n1_10 + n2_10
    ans = convert_from_base_10(ans, b)
    return ans

def subtraction(n1, b1, n2, b2, b):
    n1_10 = convert_to_base_10(n1,b1)
    n2_10 = convert_to_base_10(n1,b1)
    ans = n1_10 - n2_10
    ans = convert_from_base_10(ans, b)
    return ans

def multiplication(n1, b1, n2, b2, b):
    n1_10 = convert_to_base_10(n1,b1)
    n2_10 = convert_to_base_10(n1,b1)
    ans = n1_10 * n2_10
    ans = convert_from_base_10(ans, b)
    return ans

print subtraction('1010', 2,'100', 10, 9)
print addition('1010', 2,'100', 10, 9)
print multiplication('1010', 2,'100', 10, 9)

For the conversion to base ten, you can just use the normal int function. It takes as a second argument a base (as int). In this case the first argument is required to be a string (so it has exactly the same interface as your convert_to_base_10 function.

For the convert_from_base_10 function, I would do something closer to this answer. It is recursive, which should be fine here (unless you convert numbers with more than a thousand digits in binary, because one thousand is the default maximum recursion depth in Python). It is also a lot easier to read, IMO.

Finally, your calculation functions all proceed after the same scheme:

Convert the inputs to decimal
Calculate the result in decimal
Convert to the desired base

For this you could define a decorator that can wrap the actual calculation, which can then just assume everything is in base ten. So I would do:

def digit_to_char(digit):
    if digit < 10:
        return str(digit)
    return chr(ord('a') + digit - 10)

def str_base(number,base):
    if number < 0:
        return '-' + str_base(-number, base)
    d, m = divmod(number, base)
    if d > 0:
        return str_base(d, base) + digit_to_char(m)
    return digit_to_char(m)

def calculate_in_decimal(func):
    def wrapper(n1, b1, n2, b2, b):
        n1_10 = int(n1, b1)
        n2_10 = int(n2, b2)
        ans = func(n1_10, n2_10)
        return str_base(ans, b)

# Use it like this:
@calculate_in_decimal
def addition(n1, n2):
    return n1 + n2

# Or use the operator module:
import operator
multiplication = calculate_in_decimal(operator.mul)
subtraction = calculate_in_decimal(operator.sub)

Note that I also fixed a bug, you wrote:

n1_10 = convert_to_base_10(n1,b1)
n2_10 = convert_to_base_10(n1,b1)

Where it should be:

n1_10 = convert_to_base_10(n1,b1)
n2_10 = convert_to_base_10(n2,b2)


Find unique characters and count non unique characters We scan the string from left to right finding the unique characters. We also need to find the non unique characters and count them on how many times they appear.
If we can’t find any unique characters, then we return an empty list.

Here’s the code:

def unique_occ(the_str):
    res1 = list(the_str)
    unique_letter = []
    non_unique = []
    for i in res1:
        #there lies the issue
        #as you can see, the loop should take care of the counting.
        #But if is this is the first time that the letters appears
        #, well there is an issue.
        if i not in unique_letter:
            unique_letter.append(i)
        else:
            non_unique.append(i)

    #from that, I need to sort the letters as the for loop is not doing
    #what it is supposed to do

    unique_sort = [elts for elts in unique_letter if elts not in non_unique]
    non_unique1 = [elts for elts in unique_letter if elts in non_unique]

    #the worst part is I need to sort again the letters that have appeared more than once and concatenate them into the list again

    non_unique = non_unique + non_unique1
    non_unique2 = {i:non_unique.count(i) for i in non_unique}

    #finally, the letters have been sorted between the unique and the non unique
    return 'unique letter is {} and non unique letter is {}'.format(unique_sort,non_unique2)

The code works.

There is probably an easier way but I don't see how, at least for now. I'm unsatisfied about the pythonic way of the code.

Any help or insight will be welcomed.

A simple and pythonic way to accomplish this is using collections.Counter. It's an easy way to count elements in an iterable. Have a look:

from collections import Counter

def unique_letters(string: str) -> str:
    """
    Finds all the unique/non unique letters in the string
    """
    letters = Counter(string)

    unique = [key for key in letters if letters[key] == 1]
    non_unique = {key: letters[key] for key in letters if letters[key] > 1}
    return f"Unique: {unique} | Not Unique: {non_unique}"

This also takes advantage of f-strings, allowing you to directly implement variables into your strings. It's a personal preference, but to me it looks a lot nicer than using .format().


Project Euler#61: Cyclical figurate numbers Project Euler Problem 61 asks:

  Triangle, square, pentagonal, hexagonal, heptagonal, and octagonal numbers are all figurate (polygonal) numbers and are generated by the following formulae:

  $$\begin{array}{lll}
\textrm{Triangle}  & P_{3,n}=n(n+1)/2  & 1, 3, 6, 10, 15, \ldots\\
\textrm{Square}   & P_{4,n}=n^2  & 1, 4, 9, 16, 25, \ldots\\
\textrm{Pentagonal}  & P_{5,n}=n(3n−1)/2  & 1, 5, 12, 22, 35, \ldots\\
\textrm{Hexagonal}  & P_{6,n}=n(2n−1)  & 1, 6, 15, 28, 45, \ldots\\
\textrm{Heptagonal}  & P_{7,n}=n(5n−3)/2  & 1, 7, 18, 34, 55, \ldots\\
\textrm{Octagonal}  & P_{8,n}=n(3n−2)  & 1, 8, 21, 40, 65, \ldots
\end{array}$$

  The ordered set of three 4-digit numbers: 8128, 2882, 8281, has three interesting properties.

  The set is cyclic, in that the last two digits of each number is the first two digits of the next number (including the last number with the first).
  Each polygonal type: triangle \$(P_{3,127}=8128)\$, square \$(P_{4,91}=8281)\$, and pentagonal \$(P_{5,44}=2882)\$, is represented by a different number in the set.
  This is the only set of 4-digit numbers with this property.

  Find the sum of the only ordered set of six cyclic 4-digit numbers for which each polygonal type: triangle, square, pentagonal, hexagonal, heptagonal, and octagonal, is represented by a different number in the set.

While revising my previous solutions to bound under 5 seconds (because this is the most logical limit for any project Euler solution, otherwise the algorithm is flawed or not the best) I came to this which runs in around 10-15 seconds (peak at 12).

I have tried to make it better but I can't think of any remarkable improvements yet.

    static Stack<Integer> figurateStack;

    public static int get61() {
    figurateStack = new Stack<Integer>();
    int count = -1;
    int i = 1111;// minimum
    final int n = 6;
    final int low_lim = 3;
    int[] a = new int[n];
    boolean flag = false;// flag to use as a stepback conjugate
    boolean stepback = false;
    do {
        flag = false;
        if (count < n - 2) {
            for (; i < 9999; i++) {
                if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
                    if (count == -1 || (count >= 0 && i / 100 == a[count] % 100) && i > 1000) {
                        ++count;
                        a[count] = i;
                        break;
                    } else {
                        figurateStack.pop();
                    }
                }
            }
        } else {
            i = (i % 100) * 100 + a[0] / 100;
            if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
                ++count;
                a[count] = i;
                break;
            } else {
                stepback = true;
            }
        }
        if (i == 9999) {
            stepback = true;
        }
        if (stepback) {
            figurateStack.pop();
            i = a[count] + 1;
            --count;
            flag = true;
            stepback = false;
        }
        if (count < n - 2 && !flag) {
            i = 100 * (i % 100) + 10;
            flag = false;
        }
    } while (count < 6);
    return a[0] + a[1] + a[2] + a[3] + a[4] + a[5];
}

public static boolean isSpecialFigurate(int x, int lowlim, int upplim) {
    for (int index = lowlim; index <= upplim; index++) {
        if (!figurateStack.contains(index)) {
            long n = (long) Math.floor((Math.sqrt(8 * (index - 2) * x + (index - 4) * (index - 4)) + (index - 4)) / (2 * (index - 2)));
            if (getFigurateNumbers(n, index) == x) {
                figurateStack.push(index);
                return true;
            }
        }
    }
    return false;
}

    public static long getFigurateNumbers(long n, int r) {
    return (n * n * (r - 2) - n * (r - 4)) / 2;
}

boolean flag = false;//FLAG flag to use as a stepback conjugate
boolean stepback = false;
do {
    flag = false;//FLAG
    if (count < n - 2) {
        for (; i < 9999; i++) {
            if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
                if (count == -1 || (count >= 0 && i / 100 == a[count] % 100) && i > 1000) {
                    ++count;
                    a[count] = i;
                    break;
                } else {
                    figurateStack.pop();
                }
            }
        }
    } else {
        i = (i % 100) * 100 + a[0] / 100;
        if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
            ++count;
            a[count] = i;
            break;
        } else {
            stepback = true;
        }
    }
    if (i == 9999) {
        stepback = true;
    }
    if (stepback) {
        figurateStack.pop();
        i = a[count] + 1;
        --count;
        flag = true;//FLAG
        stepback = false;
    }
    if (count < n - 2 && !flag) {//FLAG
        i = 100 * (i % 100) + 10;
        flag = false;//FLAG
    }
} while (count < 6);
return a[0] + a[1] + a[2] + a[3] + a[4] + a[5];

You've got a flag variable in there. But from what I see, you don't need it. I've labeled all the usages with //FLAG.

The only line that sets flag to true is inside the if statement stepback. stepback is triggered by i == 9999 or some other condition earlier.

flag is set to false at the top of your do-while loop. So every iteration, it's set to false.

Because of this, the flag variable cannot be changed from false until the stepback if-statement.

So we can do this:

    if (stepback) {
        figurateStack.pop();
        i = a[count] + 1;
        --count;
        flag = true;//FLAG
        stepback = false;
    } else if (count < n - 2) {
        i = 100 * (i % 100) + 10;
        flag = false;//FLAG
    }

if stepback occured, flag is true, and the if statement thereafter becomes always false. So turn it into an else.

You don't make any other checks for flag, so now flag can be removed.

boolean stepback = false;
do {
    if (count < n - 2) {
        for (; i < 9999; i++) {
            if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
                if (count == -1 || (count >= 0 && i / 100 == a[count] % 100) && i > 1000) {
                    ++count;
                    a[count] = i;
                    break;
                } else {
                    figurateStack.pop();
                }
            }
        }
    } else {
        i = (i % 100) * 100 + a[0] / 100;
        if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
            ++count;
            a[count] = i;
            break;
        } else {
            stepback = true;
        }
    }
    if (i == 9999) {
        stepback = true;
    }
    if (stepback) {
        figurateStack.pop();
        i = a[count] + 1;
        --count;
        stepback = false;
    } else if (count < n - 2) {
        i = 100 * (i % 100) + 10;
    }
} while (count < 6);
return a[0] + a[1] + a[2] + a[3] + a[4] + a[5];

Next, we can collapse the if statement i == 9999 to be included in the stepback check:

Instead of

    if (i == 9999) {
        stepback = true;
    }
    if (stepback) {

We just do

    if (stepback || i == 9999) {

Because stepback disables itself, we don't need to worry about previous iterations of the while loop having stepback = true.

There's also an explicit else that we can make implicit...

        if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
            ++count;
            a[count] = i;
            break;
        } else {
            stepback = true;
        }

to

        if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
            ++count;
            a[count] = i;
            break;
        }
        stepback = true;

Making the total code

boolean stepback = false;
do {
    if (count < n - 2) {
        for (; i < 9999; i++) {
            if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
                if (count == -1 || (count >= 0 && i / 100 == a[count] % 100) && i > 1000) {
                    ++count;
                    a[count] = i;
                    break;
                } else {
                    figurateStack.pop();
                }
            }
        }
    } else {
        i = (i % 100) * 100 + a[0] / 100;
        if (isSpecialFigurate(i, low_lim, low_lim + (n - 1))) {
            ++count;
            a[count] = i;
            break;
        }
        stepback = true;
    }
    if (stepback || i == 9999) {
        figurateStack.pop();
        i = a[count] + 1;
        --count;
        stepback = false;
    } else if (count < n - 2) {
        i = 100 * (i % 100) + 10;
    }
} while (count < 6);
return a[0] + a[1] + a[2] + a[3] + a[4] + a[5];


Bash script to backup from S3 to GC I'm working on a script to backup S3 to GC everyday. I'm not sure if my script has any potential bug or error that might have destroyed anything from both sides? The data is quite sensitive and I don't want to mess it up. Anything is welcomed.

#!/bin/sh

if hash aws 2>/dev/null; then
    echo 'awscli is installed.'
else
    echo 'Please install awscli by running sudo pip install awscli'
    exit
fi

if hash gsutil 2>/dev/null; then
    echo 'gsutil is installed.'
else
    echo 'Please install gsutil by running sudo pip install gsutil'
fi

if env | grep -q ^BACKUP_TO_EMAIL=
then
    echo "After backup is done email will be send to $BACKUP_TO_EMAIL"
else
    echo "BACKUP_TO_EMAIL is not set please set it"
    exit
fi

if env | grep -q ^S3_BUCKET=
then
    echo "Checking if $S3_BUCKET exists"
    if aws s3 ls "s3://$S3_BUCKET" 2>&1 | grep -q 'AllAccessDisabled'
    then
        echo "bucket $S3_BUCKET doesn't exist please check again."
        exit
    fi
else
    echo 'S3_BUCKET is not set please set it'
    exit
fi

if env | grep -q ^GC_BUCKET=
then
    echo "Checking if $GC_BUCKET exists"
    if gsutil ls "gs://$GC_BUCKET" 2>&1 | grep -q 'AccessDeniedException'
    then
        echo "bucket $GC_BUCKET doesn't exist please check again."
        exit
    fi
else
    echo 'GC_BUCKET is not set please set it'
    exit
fi

echo "Backing up now..."
#`gsutil -m rsync -r s3://$S3_BUCKET gs://$GC_BUCKET`

echo "Creating new backup folder"
datestamp=$(date +%m%d%y)
mkdir $datestamp

echo "Downloading backup"
aws s3 sync "s3://$S3_BUCKET" $datestamp

echo "Compressing site backup"
tar -zcvf $datestamp.tar.gz $datestamp
file=$datestamp.tar.gz
rm -rf $datestamp

echo "Uploading to GC"
gsutil cp $file "gs://$GC_BUCKET"

echo "Deleting temporary files"
rm $file

echo "Sending email"
body="Backup is complete and the file is in gs://$GC_BUCKET/$file"

echo $body | mail $BACKUP_TO_EMAIL -s "S3 to GS backup"

Looks mostly pretty good.

When exiting with error, it's a good practice to specify a non-zero exit code, for example exit 1.

This looks odd:

if env | grep -q ^S3_BUCKET=
then
    echo "Checking if $S3_BUCKET exists"
    if aws s3 ls "s3://$S3_BUCKET" 2>&1 | grep -q 'AllAccessDisabled'

It's odd to check environment variables using env | grep.
And in this example, it looks error-prone too,
because S3_BUCKET might be defined empty, with no values.

I don't know aws, but I have a strong feeling that in case of S3_BUCKET= (empty) you would rather raise an error than run execute aws s3 ls "s3://".

I suggest to replace with a simple variable check:

if test "$S3_BUCKET"; then

In this code:

tar -zcvf $datestamp.tar.gz $datestamp
file=$datestamp.tar.gz

It would be better to set file first, and then use it in the tar

I like to avoid nested if statements when possible. For example here:

if env | grep -q ^S3_BUCKET=
then
    echo "Checking if $S3_BUCKET exists"
    if aws s3 ls "s3://$S3_BUCKET" 2>&1 | grep -q 'AllAccessDisabled'
    then
        echo "bucket $S3_BUCKET doesn't exist please check again."
        exit
    fi
else
    echo 'S3_BUCKET is not set please set it'
    exit
fi

I would flatten by inverting the outer if:

if test ! "$S3_BUCKET"
then
    echo 'S3_BUCKET is not set please set it'
    exit 1
fi

echo "Checking if $S3_BUCKET exists"
if aws s3 ls "s3://$S3_BUCKET" 2>&1 | grep -q 'AllAccessDisabled'
then
    echo "bucket $S3_BUCKET doesn't exist please check again."
    exit 1
fi


CSRF Protection in PHP I'm currently working on CSRF prevention and I would love to hear your feedback for my code. I assume that there are no XSS vulnerabilities in order for the CSRF protection to work. Furthermore, I assume that all state-changing/critical actions are done via post-requests. One token is generated per session.

Here is the code:

<?php

/*
 * CSRF prevention module
 */

class CSRF {

    private static $key = "token";

    public static function init() {
        if (session_status() === PHP_SESSION_NONE)
            die("[CSRF::init] PHP session engine is not started");

        if (!self::has_tok())
            self::set_tok(self::gen_tok());
    }

    public static function inject_token() {
        if (!self::has_tok())
            die("[CSRF::secure_form] There is no token");

        echo "<input type='hidden' name='" . self::$key
           . "' value='" . self::tok() . "'>\n";
    }

    public static function valid_post() {
        $src = parse_url(self::source_origin(), PHP_URL_HOST);
        return $src === $_SERVER['SERVER_NAME']
            && isset($_POST[self::$key])
            && self::match_tok($_POST[self::$key]);
    }

    private static function gen_tok() {
        return base64_encode(random_bytes(64));
    }

    private static function has_tok() {
        return isset($_SESSION[self::$key]);
    }

    private static function set_tok($tok) {
        $_SESSION[self::$key] = $tok;
    }

    private static function match_tok($tok) {
        return self::has_tok() && $tok === self::tok();
    }

    private static function tok() {
        return $_SESSION[self::$key];
    }

    private static function source_origin() {
        if (isset($_SERVER['HTTP_ORIGIN']))
            return $_SERVER['HTTP_ORIGIN'];
        if (isset($_SERVER['HTTP_REFERER']))
            return $_SERVER['HTTP_REFERER'];
        return "";
    }
}

?>

Usage example:

 <form action="foo.php" method="post">
     <?php CSRF::inject_token(); ?>
     ...
 </form>

And then in foo.php:

<?php
    ...
    if (!CSRF::valid_post())
        die("CSRF");
    ...
?>

Any suggestions for improvements? Did I miss anything? I know adding the tokens to forms could be automated with javascript (there is an OWASP CSRF Protection module), but I prefer to have a site that can run without javascript. I don't plan on having too many forms, so I think I can manage to not forget securing one.

I recently added checking the origin/referer after reading OWASP CSRF Prevention Cheat Sheet, but I'm not sure I did it properly.

EDIT: I just realized from the code example at OWASP that

(1) I should compare protocol and port, too, not just the host.

This seems quite reasonable as a baseline for CSRF protection. Critically, you must ensure that you do not set the Access-Control-Allow-Origin (CORS) header to a wildcard, otherwise an attacker can just perform an Ajax request to your site and gain access to the CSRF token.

A criticism of single global CSRF tokens is that that exposing the CSRF token allows re-use across different requests, which can be avoided. The upside of the single global token model is that it is very simple to implement and therefore has less chance of bugs. Additionally, it is unlikely that an attacker will gain read-access to only one page and not others, particularly as XSS would allow Ajax requests to other pages on the same origin (same domain).

If you wish to generate tokens that are locked to their initial purpose (i.e. a particular form target URI), a fairly sensible approach is to use HMAC-SHA256 to generate each token. Generate a global CSRF key as part of your session data, then for each token you need to generate compute a HMAC-SHA256 digest of the target URI using the CSRF key as the HMAC key, then inject that digest as the CSRF token in the form. For validation, take the request URI and hash it using the CSRF key, then compare that to the CSRF token provided as part of the request.

If you're building any kind of framework to handle MVC or MVVM then I would implement the CSRF protection as a by-default for all requests in the parent view or controller class, then have a way to opt-out of CSRF validation explicitly. CodeIgniter 3.x handles this quite nicely, so you should check out their code as an example.

One thing I would probably add is logging of cases where CSRF validation fails, although the way you do that is very much down to your platform.


Replacing simple jQuery methods for better use There are a few common jQuery call I find my self calling when creating my app. I need some help and maybe a better way to do all this or rewrite it.
1) Singleton Selector
If I want to select only one class I would do it like this:
$("#tempo").on("click", ".entity", function(e){
    $(".selected").removeClass("selected");
    $(this).addClass('selected');
    // ....

2) Resource Selector
I have a div with all the resource elements (divs,ul) with formated code added class and even children). This is the code I write a lot to select it, clone it and append it.
// $R = $('div.resources'); - Create only once.
$R.find(".folder").clone()
    .addClass(id++)
    .appendTo("#folders");

3) Prevent Double Actions
I have quite a lot of ajax request to the server once a button gets clicked and have to disable the button from being double clicked each time.
$("button#new_game").on("click",function(e){
    if( $(this).hasClass("clicked") == false ){
        var $pre = $(this);
            $pre.addClass("clicked");

        createAjaxCall(function(result){
           $pre.removeClass("clicked");
        }
    }
});

Adding the clicked class can allow me to customize the button a bit I should mention, So I can't really use .data()
These are the three main jQuery repetitive/non optimized problems I run into.

I'd say you're on the right track with all your code. The trick is probably to abstract some of the oft-repeated code into functions/plugins.

For the "singleton selector", wrapping your code in a plugin would seem the way to go. But note that your current code might need tweaking. You have this line:

 $(".selected").removeClass("selected")

which will remove the selected class from elements anywhere on the page. In your example, you'd probably want only to remove the class from .entity elements within the #tempo element.

Here's a (very quick) sketch of a plugin, that should do what you want (and keep it limited to the container element):

$.fn.singleSelection = function(classSelector, handler) {
    return this.each(function() {
        var container = $(this);
        container.on("click", classSelector, function (event) {
            container.find(classSelector + ".selected").removeClass("selected");
            $(this).addClass("selected");
            if( typeof handler === "function" ) {
                handler.call(this, event);
            }
        });
    });
};

Your code can then be shortened to

$("#tempo").singleSelection(".entity"); // you can add a click handler too

Here's a demo.

But... jQueryUI already has this functionality, so if you feel like using that, go right ahead. It's certainly been tested better than what you see above :)
However, for such a simple thing, jQueryUI seems like overkill. But there are probably other, leaner plugins you can find, which'll do what you need - this is just an example.

For your second point, I'd probably wrap the code in an addFolder function somewhere. But it's a little hard to be more specific without knowing more about the context.

For your third point, you could again make a simple jQuery plugin that'd only forward events to your click handler if the element doesn't have the "clicked" class. There are also various "debounce" plugins out there to throttle clicks, but most simply impose a time limit rather than explicitly wait for an ajax operation to finish.

In any event, here's another (very quick) sketch of a jQuery plugin

$.fn.clickOnce = function(handler) {
    return this.each(function() {
        var element = $(this);
        element.on("click", function (event) {
            if( !element.hasClass("clicked") ) {
                element.addClass("clicked");
                if( typeof handler === "function" ) {
                    handler.call(this, event);
                }
            }
        });
    });
};

In your code you could then do:

$("button#new_game").clickOnce(function (event) {
    // this will only be called if the button wasn't already clicked
    var button = $(this);
    createAjaxCall(function (result) {
        // it's your responsibility to remove the "clicked" class
        button.removeClass("clicked");
    });
});

Here's a demo

Again, there are no doubt ready-made plugins that do this and do it better.

By the way, I personally find it easier to pass jQuery's XHR objects around, as they act as deferred objects/promises. I.e.:

var xhr = $.ajax(...); // create an XHR obj from any of jQuery's ajax functions

xhr.done(function (data) {
  // success handler
});

xhr.fail(function (err) {
  // failure handler
});

xhr.always(function () {
  // complete handler
});

So if your createAjaxCall() function returns an xhr object, you can attach a handler to the always "event", and use that to clear the clicked class. That seems more robust (and readable) to me.

p.s. You could use a data-* attribute (set with .attr() instead of .data()), and still have custom styling with CSS like

button[data-clicked='true'] { // CSS attribute selectors are neat!
  ...
}

but browser support is limited compared to using simple classes.


Note: The code examples below split lines on 1+ whitespace character instead of tabs - perhaps the way it was pasted in your question converted tabs to spaces...
Removing the array of diffs
Because the overall equation for this problem would be a summation of differences, the two calls to .reduce() can be combined into one- instead of having one function to essentially push a difference into an array and then a separate function to sum those differences, just add the difference to the cumulative total. In addition to reducing the number of function calls by the number of lines, the elimination of the array of differences should drastically reduce memory consumption.
const CumulativeDiffs = (totalSoFar, val) => {
    const row = val.split(/\s+/);
    const max = Math.max(...row);
    const min = Math.min(...row);
    return totalSoFar + max - min; //add diff to cumulative total
};

const solution = INPUT.split(/\n/)
    .reduce(CumulativeDiffs, 0);

const INPUT =
  `6046   6349    208 276 4643    1085    1539    4986    7006    5374    252 4751    226 6757    7495    2923
1432    1538    1761    1658    104 826 806 109 939 886 1497    280 1412    127 1651    156
244 1048    133 232 226 1072    883 1045    1130    252 1038    1022    471 70  1222    957
87  172 93  73  67  192 249 239 155 23  189 106 55  174 181 116
5871    204 6466    6437    5716    232 1513    7079    6140    268 350 6264    6420    3904    272 5565
1093    838 90  1447    1224    744 1551    59  328 1575    1544    1360    71  1583    75  370
213 166 7601    6261    247 210 4809    6201    6690    6816    7776    2522    5618    580 2236    3598
92  168 96  132 196 157 116 94  253 128 60  167 192 156 76  148
187 111 141 143 45  132 140 402 134 227 342 276 449 148 170 348
1894    1298    1531    1354    1801    974 85  93  1712    130 1705    110 314 107 449 350
1662    1529    784 1704    1187    83  422 146 147 1869    1941    110 525 1293    158 1752
162 1135    3278    1149    3546    3686    182 149 119 1755    3656    2126    244 3347    157 865
2049    6396    4111    6702    251 669 1491    245 210 4314    6265    694 5131    228 6195    6090
458 448 324 235 69  79  94  78  515 68  380 64  440 508 503 452
198 216 5700    4212    2370    143 5140    190 4934    539 5054    3707    6121    5211    549 2790
3021    3407    218 1043    449 214 1594    3244    3097    286 114 223 1214    3102    257 3345`;

const CumulativeDiffs = (totalSoFar, val) => {
  const row = val.split(/\s+/);
  const max = Math.max(...row);
  const min = Math.min(...row);
  return totalSoFar + max - min;
};

const solution = INPUT.split(/\n/)
  .reduce(CumulativeDiffs, 0);
console.log("solution ", solution);

Sorting
I did consider sorting the array of values in each row using Array.sort(), which works, but the values all need to be converted to integers first (e.g. using parseInt(), or just a function that returns the number added to 0), which slows things down a little. But with that change there is no need to call Math.max() and then Math.min() - simply sort the numbers, then add the last element to the cumulative total and subtract the first element.
const CumulativeDiffs = (totalSoFar, val) => {
  //let row = val.split(/\s+/).map((num) => +num); //still slower than finding min and max
  let row = val.split(/\s+/).map((num) => parseInt(num, 10));
  row.sort((a, b) => a - b); //sort numbers in each row
  return totalSoFar + row[(row.length - 1)] - row[0]; // add difference
};

const INPUT =
  `6046   6349    208 276 4643    1085    1539    4986    7006    5374    252 4751    226 6757    7495    2923
1432    1538    1761    1658    104 826 806 109 939 886 1497    280 1412    127 1651    156
244 1048    133 232 226 1072    883 1045    1130    252 1038    1022    471 70  1222    957
87  172 93  73  67  192 249 239 155 23  189 106 55  174 181 116
5871    204 6466    6437    5716    232 1513    7079    6140    268 350 6264    6420    3904    272 5565
1093    838 90  1447    1224    744 1551    59  328 1575    1544    1360    71  1583    75  370
213 166 7601    6261    247 210 4809    6201    6690    6816    7776    2522    5618    580 2236    3598
92  168 96  132 196 157 116 94  253 128 60  167 192 156 76  148
187 111 141 143 45  132 140 402 134 227 342 276 449 148 170 348
1894    1298    1531    1354    1801    974 85  93  1712    130 1705    110 314 107 449 350
1662    1529    784 1704    1187    83  422 146 147 1869    1941    110 525 1293    158 1752
162 1135    3278    1149    3546    3686    182 149 119 1755    3656    2126    244 3347    157 865
2049    6396    4111    6702    251 669 1491    245 210 4314    6265    694 5131    228 6195    6090
458 448 324 235 69  79  94  78  515 68  380 64  440 508 503 452
198 216 5700    4212    2370    143 5140    190 4934    539 5054    3707    6121    5211    549 2790
3021    3407    218 1043    449 214 1594    3244    3097    286 114 223 1214    3102    257 3345`;

const CumulativeDiffs = (totalSoFar, val) => {
  //let row = val.split(/\s+/).map((num) => +num); //still slower than finding min and max
  let row = val.split(/\s+/).map((num) => parseInt(num, 10));
  row.sort((a, b) => a - b);
  return totalSoFar + row[(row.length - 1)] - row[0];
};

const solution = INPUT.split(/\n/)
  .reduce(CumulativeDiffs, 0);

console.log("solution ", solution);

Compare these approaches with yours in this jsPerf.
Naming Things
Like @Igor mentioned in this answer to one of your previous posts the variable names could changed to better describe the values, and functions could better describe what they do. For example - summer might be a better name than sum, since it provides a sum of values.


Read Space Delimited Text File to Standardized Data Type From the README.md
Introduction:
The purpose of data-in is to read all the lines from a text file and return the file's contents as a Clojure value in a standardized manner.
Specification:
Takes:

Filename

Assumes:

a space delimited text file

Returns:

Vector [L1,L2,...Ln] where Li:

represents a line in Filename
is a vector
contains EDN values

Example:
If my-file.txt contains:
3 9
5 4
6 5
3 2

then (data-in "./my-file.txt") returns:
[[3 9] [5 4] [6 5] [3 2]]
Comments:

To avoid executing arbitrary code, data-in reads values from files with clojure.edn/read-string.

The input is not read lazily. The assumption being that sorting or an equivalent operation will be performed. The basis for this assumption is the proximate use case: reading data files for Discreet Optimization @ Coursera.

The private function f is called f to match the argument f at the call site in the private function get-data. This reflects a personal preference for mathematical notation when invoking mathematical abstractions.

Code:
(ns data-in.core
  (:require [clojure.java.io :as io]
            [clojure.edn :as edn]))

(defn- get-data
  [data-file f]
  (with-open [rdr (clojure.java.io/reader data-file)]
        (doall (map f (line-seq rdr)))))

(defn- f
  [line]
  (vec (map (fn [x] (edn/read-string x))
            (clojure.string/split line #" "))))

(defn data-in
  [filename]
  (vec (get-data filename f)))

First off, I'm going to have to complain about your taste in function names, which will probably be unproductive and just lead to hurt feelings on both sides. Yes, it's that function f. I don't mind so much if small inner functions defined with letfn have names like f, but to my eye, something declared at the top level of namespace—private or no—should have a more descriptive name. I would probably not define f as a top-level function at all; see below for more on that. It's also confusing that you call the top-level function f, then also call the function passed as a parameter to get-data by the name f and then go ahead and pass f as the parameter f in data-in.

I kind of don't like that you expect the output of data-in to be a vector of vectors, but you depend on the function passed as a parameter to get-data to return a vector instead of putting that logic inside data-in. If I had a top-level private function that's only used once and seems too specific to ever be useful again, like f, I would consider making it an anonymous function that gets passed in the one time it's used. So I would probably rewrite data-in like this:

(defn data-in
    [filename]
    (vec (get-data filename #(vec (map edn/read-string
                                       (clojure.string/split % #" ")))

Or, if you're uncomfortable with the shortcut syntax for anonymous functions (it can get pretty hairy), like this:

(defn data-in
    [filename]
    (vec (get-data filename (fn [line] (vec (map edn/read-string
                                             (clojure.string/split line #" ")))

Now data-in is longer and a bit uglier. But this way, all the logic for data-in is in data-in. You expected data-in to return a vector of vectors, but as it was before, data-in relied on f processing each line into a vector, so the logic of one function was spread across multiple functions.

Notice how the whole thing is now separated into a general portion, the get-data function, which could be useful again later on when you need to read some lines from a file, process them in some way, and return them in a sequence; and a specific portion, data-in, that contains all that bits which are only relevant to the problem you have right now.

Pursuant to that, I don't really like that get-data is private. It seems like a perfectly good, general, widely useful function. For example, suppose we have a CSV file with columns a, b, and c, separated by spaces. Then we could write:

(get-data "MyCSV.csv" #(let [{a 0, b 1, c 2} (clojure.string/split % #" ")]
                            {:a a, :b b, :c c}))

and instead of a sequence of vectors, we'd get back a sequence of maps with keys :a, :b, and :c. This usage is totally unrelated to your original intention for this code, but it shows how generally useful the function is—so why make it private and hide it where no one can access it? Let it out and see what uses people can find for it.

Finally, for such a small program it probably doesn't matter, but one of my favorite features of Clojure is docstrings. I would write one for any function I was publicly exposing:

(defn data-in
    "Returns a vector of vectors of elements from the given file."
    [filename]
    (vec (get-data filename #(vec (map edn/read-string
                                       (clojure.string/split % #" ")))

Then anyone can call up a REPL, load your namespace, and type (doc data-in) to read the documentation.


Displaying user scores I have started teaching myself MySQL and created a simple application which asks for user input stores it and displays it. Can you guys please let me know if there is something I am doing wrong or any tips on what can be improved? I want to build a student management application which will use a MySQL database and want to make sure I know the basics.

I have a class that holds connection method and closing methods:

import java.sql.*;

public class DBUtil {

// Connecting to database.
public static Connection getConnection() throws SQLException {
String url = "jdbc:mysql://localhost:3306/database";
String user = "root";
String pass = "admin";

Connection connection = DriverManager.getConnection(url, user, pass);
    return connection;
}

// Close database connection.
public static void closeConnection(Connection con) {
    try {
        if(con != null) {
            con.close();
            System.out.println("Database Connection Closed!");
        }
    }catch(Exception e) {
        e.printStackTrace();
    }
}
// Close Prepared Statement.
public static void closePrepStatement(PreparedStatement pStmt) {
    try {
        if(pStmt != null) {
            pStmt.close();
            System.out.println("Prepared Statemet Closed");
        }
    }catch (Exception e) {
        e.printStackTrace();
    }
}
// Close Result Set.
public static void closeResSet(ResultSet rSet) {
    try{
        if(rSet != null) {
            rSet.close();
            System.out.println("Result Set Closed");
        }
    }catch (Exception e) {
        e.printStackTrace();
    }
}
}

This class gets the input, stores it, and displays it:

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.Scanner;

// SQL variables defined
public class Source {
DBUtil dbConnection = new DBUtil();
Scanner input = new Scanner(System.in);
Connection connection = null;
PreparedStatement pStatement = null;
ResultSet rSet = null;

// Constructor
public Source() {
    getInput();
    displayData();
}
// Getting user input and saving to database
private void getInput() {

    System.out.println("What is your name?");
    String fName = input.nextLine();
    System.out.println("What is your second name?");
    String sName = input.nextLine();
    System.out.println("What is your proffesion?");
    String userProff = input.nextLine();

    try {
         connection = DBUtil.getConnection();
         String sqlQuery = "INSERT INTO employees_info (firstname,  secondname, proffesion) VALUES (?, ?, ?)";
         pStatement = connection.prepareStatement(sqlQuery);
         pStatement.setString(1, fName);
         pStatement.setString(2, sName);
         pStatement.setString(3, userProff);
         pStatement.executeUpdate();
         System.out.println("Record saved!");

    }catch(Exception e) {
        e.printStackTrace();
    }finally {
        DBUtil.closePrepStatement(pStatement);
        DBUtil.closeConnection(connection);
    }
}
// Get data from database and display
private void displayData() {
    System.out.println("Current records from the database:");

    try {
        connection = DBUtil.getConnection();
        String sqlQuery = "SELECT * FROM employees_info";
        pStatement = connection.prepareStatement(sqlQuery);
        rSet = pStatement.executeQuery();
        while(rSet.next()) {
            System.out.println("");
            System.out.println("Next Record:");
            System.out.println(rSet.getInt("ID"));
            System.out.println(rSet.getString("firstname"));
            System.out.println(rSet.getString("secondname"));
            System.out.println(rSet.getString("proffesion"));
        }
    }catch (Exception e) {
        e.printStackTrace();
    }finally {
        DBUtil.closeResSet(rSet);
        DBUtil.closePrepStatement(pStatement);
        DBUtil.closeConnection(connection);
    }
}

}

It is usually considered bad practice to use * imports.  Be specific about your dependencies.
Don't use root DB access for your application.  That should truly be reserved for a root DB user.  Create a MySQL user for the application with appropriate privilege levels.
Don't hard code your DB credentials into your code.  This should ideally be derived from application or environmental configuration.
Should you really be swallowing up your exceptions like this?
Your validation against NPE's seems odd here. Rather than a pattern like

this:

try {
    if(con != null) {
        con.close();
        System.out.println("Database Connection Closed!");
    }
}catch(Exception e) {
    e.printStackTrace();
}

perhaps you should be doing something like:

if(con == null) {
    throw new NullPointerException('...');
}
// rest of code in method. Now, without need to be nested in conditional

But you should definitely let the caller know if they passed you a null pointer vs. swallowing it.

I find it very odd to be opening prepared statements and result sets in your Source class but closing them using the DBUtil class. This seems very asymmetric to me and a leaking of responsibility outside the class.
Don't get yourself into the bad habit of using SELECT * queries.  They can be wasteful of bandwidth (by pulling unnecessary fields from DB into application), they can obfuscate the data structure that is being worked in within the code (i.e. you have to go look at the table in the database to understand what fields you are working with), and it can make your code fragile to table schema changes
getInput() is a bad method name.  This method is actually both reading in user input and inserting data in the database, something that is not implied from the method name, and activities that should probably be split into individual methods.  The same could be said about displayData(). This method is not just displaying data, but also retrieving it from the database.
Consider passing the database connection and the scanner to the Source class as dependencies rather than having the class have to hold the knowledge on how to set them up. Of course, this might require a larger thought shift towards dependency injection throughout other areas of code.
I know this is a bit of a trivial example, but why would the constructor for the class trigger both the input and ultimate display of content?
I don't see the need to have class properties for storing the prepared statement and result set objects.
You don't use dbConnection class property at all. There is probably no need for even storing the connection if you aren't going to reference if across different methods - something not happening the way this class is currently designed.
I see no need to actually connect twice to the database here, especially considering this class is really designed for the purpose of doing one single operation that is really more procedural in nature.
When building classes in the future, begin to think about how to separate class functionality from end user messaging or view output.  Again, I know this is a simple example, but the more you do this in the future, the more you are better able to make your classes re-usable..


Partial application in C++ Here is my code:

#include <iostream>
#include <type_traits>
#include <functional>
#include <tuple>

namespace meta {

template <bool B, bool... Bs>
struct all_true {
   enum : bool { value = B && all_true<Bs...>::value };
};

template <bool B>
struct all_true<B> {
    enum : bool { value = B };
};

template <typename Fn, typename... Args>
struct is_partial_t_nothrow_ctor {
    enum : bool {
        a = std::is_nothrow_constructible<Fn, Fn&&>::value,
        b = all_true<std::is_nothrow_constructible<Args, Args&&>::value...>::value,
        value = a && b
    };
};

}

template <typename Fn, typename... Args>
class partial_t {
private:
    Fn fn_;
    std::tuple<Args...> args_;

    template <std::size_t... Is, typename... ExArgs>
    decltype(auto) op_parens_impl(std::index_sequence<Is...>, ExArgs&&... ex_args) {
        fn_(std::forward<Args>(std::get<Is>(args_))...,
            std::forward<ExArgs>(ex_args)...);
    }

    template <typename... ExArgs>
    decltype(auto) op_parens_impl(ExArgs&&... ex_args) {
        return op_parens_impl(
            std::index_sequence_for<Args...>(), std::forward<ExArgs>(ex_args)...);
    }

public:
    partial_t(Fn&& fn, Args&&... args)
        noexcept(meta::is_partial_t_nothrow_ctor<Fn, Args...>::value)
        : fn_(std::forward<Fn>(fn)), args_(std::forward_as_tuple(args...))
    {}

    template <typename... ExArgs>
    decltype(auto) operator()(ExArgs&&... ex_args)
        // noexcept(?)
    {
        return op_parens_impl(std::forward<ExArgs>(ex_args)...);
    }
};

template <typename Fn, typename... Args>
auto partial(Fn&& fn, Args&&... args)
    noexcept(meta::is_partial_t_nothrow_ctor<Fn, Args...>::value)
{
    return partial_t<Fn, Args...>(std::forward<Fn>(fn), std::forward<Args>(args)...);
}

Test code:

void test(int x, int y, int z) {
    std::cout << "my first number is " << x << std::endl;
    std::cout << "my second number is " << y << std::endl;
    std::cout << "my favorite number is " << z << std::endl;
}

int main() {
    auto f = partial(test, 5, 3);
    f(7);
}

which outputs:

my first number is 5
my second number is 3
my favorite number is 7

The function partial takes a function object as its first parameter and then a number of arguments. It returns a function object which binds these arguments to the input function as its first parameters, then the rest of the arguments are provided later when this partial function is invoked.

Looks like you are trying to re-invent std::bind.

#include <functional>
#include <iostream>

void test(int x, int y, int z) {
    std::cout << "my first number is " << x << std::endl;
    std::cout << "my second number is " << y << std::endl;
    std::cout << "my favorite number is " << z << std::endl;
}

int main() {
    // Add placeholders for non bound values.
    using namespace std::placeholders;

    // Note the last argument is a place holder.
    auto f = std::bind(test, 5, 3, _1);

    // Now you can call and pass a placeholder argument.
    f(7);
}

As a side note: This is the kind of reason why underscore is reserved for the implementation to allow for little shortcuts like this without having to re-invent the language.

Rather than using recursive templates. I would have used an iterative aproach. With constexpr you can do quite a lot that is evaluated at compile time.

struct all_true
{
    template<typename... T>
    static bool constexpr isTrue(T... b)
    {
        bool result = true;
        for(auto const& a: {b...})
        {
            result = result && a;
        }
        return result;
    }
};

// Then your code looks like this:
b = all_true::isTrue(std::is_nothrow_constructible<Args, Args&&>::value...),


Backup a SQLite database I want to backup a single SQLite database daily up to 30 days back, but I also want to keep at least 2 backups at all times (i.e. if there have been no backups in the last 30 days because the database didn't change, I don't want to delete old backups).

I came up with this simple script that is supposed to run as a daily cronjob:

#!/bin/bash

BACKUP_FILE="/path/to/db.sqlite3"
BACKUP_DIR="$HOME/backups"

today=`date "+%Y-%m-%d"`

# Less than 31 days old, i.e. 30 days or younger
if find "$BACKUP_FILE" -type f -mtime -31 | grep -q .
then
        find "$BACKUP_DIR" -type f -mtime +30 -exec rm {} \;
fi

last_backup=$(ls -t "$BACKUP_DIR" | head -1)
if [ -n "$last_backup" ] && diff "$BACKUP_FILE" "$BACKUP_DIR/$(ls -t "$BACKUP_DIR" | head -1)" >/dev/null
then :
else
        cp "$BACKUP_FILE" "$BACKUP_DIR/$today.sqlite3"
fi

I'm not sure if using a nop in the second then clause makes sense, but it seemed cleaner to me than wrapping the condition in a test and checking $?.

The database is used by <20 people and not changed very often, but I'm not sure if I should lock the database anyways - and I'm also not sure how to lock it from a shell script.

Since this is the first bash script I've ever written for serious use, I'd appreciate any feedback on how it could be improved.

Don't use the obsolete syntax `...`

Use the modern syntax $(...):

today=$(date "+%Y-%m-%d")

Simplify file deletion

Instead of deleting files with -exec rm {} \;, a simpler way is using -delete:

find "$BACKUP_DIR" -type f -mtime +30 -delete

Don't use the obsolete syntax head -N

head -1 is obsolete syntax. Use head -n 1 instead.

Don't repeat yourself

On these lines, $(ls -t "$BACKUP_DIR" | head -1) is repeated:

last_backup=$(ls -t "$BACKUP_DIR" | head -1)
if [ -n "$last_backup" ] && diff "$BACKUP_FILE" "$BACKUP_DIR/$(ls -t "$BACKUP_DIR" | head -1)" >/dev/null

The second use should be replaced with $last_backup.

Compare files using cmp

Instead of diff, it's more efficient to compare files using cmp:

if [ -n "$last_backup" ] && cmp -s "$BACKUP_FILE" "$BACKUP_DIR/$last_backup"

Avoid empty statements

Instead of the empty then clause of the final condition,
it's better to invert it (using DeMorgan's law):

if [ ! "$last_backup" ] || ! cmp -s "$BACKUP_FILE" "$BACKUP_DIR/$last_backup"
then
        cp "$BACKUP_FILE" "$BACKUP_DIR/$today.sqlite3"
fi

Delay operations until really needed

The today variable is only used once at the end,
when certain conditions are fulfilled.
It would be better to create it right before it's needed.


Filtering array by list of keys in PHP 5.5 I have an associative array with data (let's say language codes and descriptions) and a second array with allowed keys (lang codes). I want to filter the data array by these allowed keys.

The problem is I'm bound to PHP 5.5 and I can't use ARRAY_FILTER_USE_KEY flag.

I came up with the following solution:

$langs = [
 'en' => "English",
 'de' => "German",
 'fr' => "French",
 'ru' => "Russian",
];

$allowed_langs = ['en','de'];

var_export(
array_map( function($lang) use($langs) {return $langs[$lang];} , array_combine($allowed_langs, $allowed_langs))
);

/* Output as expected:
array (
  'en' => 'English',
  'de' => 'German',
)
*/

I wonder is there a more elegant and shorter solution to this task?

I always believe that you should use built in functions wherever possible, as opposed to recreating PHP functionality with loops etc. The main reasons for saying this are that:

We should trust that PHP functions achieve their desired result in an efficient manner, and:
If they are improved in future versions your code doesn't need to change, but just gets better.

That being said, why not something like this:

$matches = array_intersect_key($langs, array_flip($allowed_langs));
var_export($matches);


DigitalOcean dynamic DNS update script Inspired by a shell script I saw on Github, I put together a Python version to update the dynamic IP address for a subdomain I have that uses DigitalOcean's nameservers. I added a check to see if the IP address actually needs to be updated, and not do the update if the IP address hasn't changed. All of the user-configurable variables - API token, domain, subdomain - are stored in a separate .env file alongside the script.

#!/usr/bin/env python3

# Import required modules
import dotenv
import json
import os
import requests

# Load user-configured variables from .env file
dotenv.load_dotenv()
token = os.environ.get('DO_API_TOKEN')
domain = os.environ.get('DO_DOMAIN')
subdomain = os.environ.get('DO_SUBDOMAIN')

# Other variables
check_ip_url = 'https://api.ipify.org'
do_api_url = 'https://api.digitalocean.com/v2/domains/'

# Get the current external IP
def get_current_ip():
    curr_ip = requests.get(check_ip_url).text.rstrip()
    return curr_ip

# Get the current subdomain IP
def get_sub_info():
    headers = {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer ' + token
    }
    response = requests.get(do_api_url + domain + '/records', headers=headers).text
    records = json.loads(response)
    for record in records['domain_records']:
        if record['name'] == subdomain:
            subdomain_info = {
                'ip': record['data'],
                'record_id': record['id']
            }
            return subdomain_info

# Update DNS records if required
def update_dns():
    current_ip_address = get_current_ip()
    subdomain_ip_address = get_sub_info()['ip']
    subdomain_record_id = get_sub_info()['record_id']
    if current_ip_address == subdomain_ip_address:
        print('Subdomain DNS record does not need updating.')
        return
    else:
        headers = {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer $token',
        }
        data = '{"data":"' + current_ip_address + '"}'
        response = requests.put(do_api_url + domain + '/records/' + subdomain_record_id, headers=headers, data=data)
        if '200' in response:
            print('Subdomain IP address updated to ' + current_ip_address)
        else:
            print('IP address update failed with message: ' + response.text)
        return

if __name__ == '__main__':
    update_dns()

It works, but isn't the prettiest Python code out there, and I'm sure if could be made a bit more efficient. Are there any stylistic changes I should make to better adhere to best practices? I have run it through a PEP8 checker, and the only thing it brought up was about the line length in a few places, which I'm not super concerned about.

#!/usr/bin/env python3

import os

import dotenv
import requests

dotenv.load_dotenv()
token = os.environ['DO_API_TOKEN']
domain = os.environ['DO_DOMAIN']
subdomain = os.environ['DO_SUBDOMAIN']

records_url = f'https://api.digitalocean.com/v2/domains/{domain}/records/'
session = requests.Session()
session.headers = {
    'Authorization': 'Bearer ' + token
}

def get_current_ip():
    return requests.get('https://api.ipify.org').text.rstrip()

def get_sub_info():
    records = session.get(records_url).json()
    for record in records['domain_records']:
        if record['name'] == subdomain:
            return record

def update_dns():
    current_ip_address = get_current_ip()
    sub_info = get_sub_info()
    subdomain_ip_address = sub_info['data']
    subdomain_record_id = sub_info['id']
    if current_ip_address == subdomain_ip_address:
        print('Subdomain DNS record does not need updating.')
    else:
        response = session.put(records_url + subdomain_record_id, json={'data': current_ip_address})
        if response.ok:
            print('Subdomain IP address updated to ' + current_ip_address)
        else:
            print('IP address update failed with message: ' + response.text)

if __name__ == '__main__':
    update_dns()

The script will not work if any variables are missing, so use [...] instead of .get(...) to throw an error ASAP if needed.
The DO URLs always start with /{domain}/records/ so I included that in the top level constant.
A requests.Session makes multiple requests to the same domain faster as it keeps the connection open, and it lets you specify info like headers once.
A few times you create a variable and immediately return it. You can just return the expression directly.
I felt that the check_ip_url constant didn't add anything, so I inlined it. This is mostly a preference.
Most of the comments do not help readers in any way, so I removed them. But if you want to describe what a function does, use a docstring.
Calling .json() on a response parses the text as JSON for you.
Moving the values of one dict to another dict before finally extracting them just adds another layer, so I just returned record directly.
You called get_sub_info() twice which meant two identical requests. To speed things up, I extracted a variable.
Again, requests makes JSON easy to use, now with the json= argument. This both converts the dictionary to a JSON string and sets the content type. But even without this, you really should have been using json.dumps rather than string concatenation.
response.ok is typically how you check if a request succeeded, or by checking the value of response.status_code. I've never seen anyone using the in operator on a response.
There's no reason to return when a function is ending anyway.