jQuery delegate events on dynamic elements with a click away handler Just thought I'd share this and see what kind of improvements could be made that I'm not aware of and taking advantage of.

The situation is this: I have a list of items on a page that have a button that opens a menu. I need a click away event as well as close others when a new menu is opened. Also some of the menu actions will replace items in the list which means I have new dynamic menu buttons on the page.

To do this efficiently (as far as I can tell) you need to use event delegation and it ends up looking like this:

$(document).delegate( '.button', 'click', function(e){
    var $button = $(this);
    var $menu = $button.closest('.menu');
    e.stopPropagation();

    // Close others
    $('.button').each(function(){
      $(this).closest('.menu').removeClass('open');
    });

    // Toggle target open/close
    if( $menu.hasClass('open') ){
        $menu.removeClass('open');
    }
    else{
        $menu.addClass('open');
    }
    $(document).on('click', function(e){
        var $target = $(e.target);
        if( !$target.is( $( $menu.find('button') ) ) ){
            if( $menu.hasClass('open') ){
                $menu.removeClass('open');
            }
        }
    });
});

Maybe there's a better way?

I believe there is no better way. Regardless, we can shorten the code a little. Also note that .delegate() has been superceded by .on() since jQuery 1.7, so I replaced your delegate with on.

$(document).on( 'click', '.button', function(e){
    var $button = $(this);
    var $menu = $button.closest('.menu');
    e.stopPropagation();

    // Close others-assumed you want all menus closed
    $('.menu').not($menu).removeClass('open');

    // Toggle target open/close
    $menu.toggleClass('open');

    $(document).on('click', function(e){
        //This function will only run when e.target isn't a button
        //Your $target.is($menu.find('button')) will always be false

        if( $menu.hasClass('open') ){
            $menu.removeClass('open');
        }
    });
});

If this does not do the same thing, you really need to share your html structure so we can get a better idea of your mark-up.

Edit
To work around the fact that the document click event is assigned every time you click on a button, you could do something like the following:

$(document).ready(function() {
   var $menu;
   $(document).on( 'click', '.button', function(e){
      var $button = $(this);
      $menu = $button.closest('.menu');
      e.stopPropagation();
      $('.menu').not($menu).removeClass('open');
      $menu.toggleClass('open');
   });
   $(document).on('click', function(e){
      if($menu && $menu.hasClass('open') ){
            $menu.removeClass('open');
        }
   });
)}


An interesting use of the reduce function to convert tab-delimited data into JSON I am learning functional programming in JavaScript. This is some code from a video tutorial by funfunfunction. I am particularly concerned about the way the existence of the customer key is checked in the customers object.

Is it weird to do it this way? Is there any other way? Also, if you switch the order of the OR statement operands, the object is overwritten. Does order matter in an OR statement?

The function:

var fs = require('fs')

var output = fs.readFileSync('data2.txt', 'utf8')
  .trim()
  .split('\r\n')
  .map(function(line) {
    return line.split('\t')
  })
  .reduce(function(customers, line) {
    customers[line[0]] = customers[line[0]] || []
    customers[line[0]].push({
      item: line[1],
      quantity: line[2],
      price: line[3],
    })
    return customers
  }, {})

console.log('output', JSON.stringify(output, null, 2));

The data:

Marry Poppins red bicycle 80  2
Marry Poppins glass vase  20  8
Abe Lincoln   gold ring   1   100
Marry Poppins umbrella    5   50
Steve Rogers  video camera    1   100
Abe Lincoln   president chair 1   1000
Marry Poppins blue dress  2   200

So for your first question, that is not a weird way to check the existence, and it is used quite frequently to check whether certain variables/properties are set (such as checking browser compatibility, optional arguments with defaults in functions). It typically just works as a "use this value if it exists, otherwise use this value."

You could check/set it any number of ways, like the ternary operator.

customer[line[0]] = customer[line[0]] ? customer[line[0]] : {};

In the same vein, you could just use an if statement to check if it is defined.

Order definitely does matter in an OR statement. In javascript (and probably most other languages) the first condition/operation is always performed, and the 2nd condition or operation is ignored unless the first condition/operation equated to a falsy value. In your example, customer[line[0]] is only assigned as an empty object if customer[line[0]] is undefined. If you reverse the order, customer[line[0]] will always be assigned to the empty object, and the 2nd half of the statement is ignored.

Javascript is the only language I've used where OR statements are normally used outside of actual control statements, so it was definitely puzzling to look at these statements at first.

Another, similar operator is the comma , operator, which is essentially the lowest in the order of operations. It just returns the last of the comma-separated statements, but all of the statements are performed.


Pseudo-random number generator port from C C code:

uint32_t pcg32_random_r(pcg32_random_t* rng)
{
  uint64_t oldstate = rng->state;
  rng->state = oldstate * 6364136223846793005ULL + rng->inc;
  uint32_t xorshifted = ((oldstate >> 18u) ^ oldstate) >> 27u;
  uint32_t rot = oldstate >> 59u;
  return (xorshifted >> rot) | (xorshifted << ((-rot) & 31));
}

Attempted Java port (slightly fixed from GitHub):

public int nextInt() {
  long oldState = state;
  state = oldState * MULTIPLIER + inc;
  int xorShifted = (int) (((oldState >>> 18) ^ oldState) >>> 27);
  int rot = (int) (oldState >>> 59);
  return Integer.rotateRight(xorShifted, rot);
}

Are there differences in behavior? * and + should give the same bitwise results for signed arithmetic in Java and unsigned in C++, >>> in Java is the same as >> in C... But am I missing something?

I cannot see any functional issues when comparing the C code against the Java code. it would seem they should produce the same results given the same input. Of course, a logical course of action would be to write a test system for the code which wraps the C code in to a program which accepts a single seed value as an argument, and produces a stream of int values in 'network byte' order. You then pipe that stream in to a Java program which produces the same stream, and compares the results. I would actually set up a test where the Java program starts a sub-process for the C version, and runs it with many input seeds, including the logical edge-cases (0, 1, -1, Long.MAX_VALUE, Long.MinValue, and all seeds from -100 to +1000). For each run, compare the first 10,000 generated values.

That would give me some measure of confidence that the code is a faithful reproduction.

Like you, I believe the Java code is a fair reproduction, but, like you, I would want to check it again.

Now,about the code, you are doing unnecessary work. There is no need to store the oldState:

public int nextInt() {
  long oldState = state;
  state = oldState * MULTIPLIER + inc;
  int xorShifted = (int) (((oldState >>> 18) ^ oldState) >>> 27);
  int rot = (int) (oldState >>> 59);
  return Integer.rotateRight(xorShifted, rot);
}

That code can be reduced to:

public int nextInt() {
  int xorShifted = (int) (((state >>> 18) ^ state) >>> 27);
  int rot = (int) (state >>> 59);
  state = state * MULTIPLIER + inc;
  return Integer.rotateRight(xorShifted, rot);
}

Also, while your comments about the >>> vs >> are accurate, and I prefer the use of >>> here, you should also know that the only shift which has a significant impact (where using >>> makes a difference vs. >>) is the part:

(state >>> 18)

In that shift, if it was a negative state, and you used >>, it would shift the sign bit to position 46 (and all higher bits). Then the subsequent >>> 27 would shift those effects in to the low-order 32-bits. Since the value is truncated to 32 bits, any sign-shift effects which happen above 32 bits are irrelevant, and as I say, the only sign-shift which can effect the low 32 bits is the first one.

Also, Java uses a mask on the int rotate value, and for Integer.rotateRight it only considers the low-5 bits in the value, so the following are identical because the rotate effectively applies the mask 0x1f to the value anyway:

Integer.rotateRight(xorShifted, (int)(oldState >>> 59))
Integer.rotateRight(xorShifted, (int)(oldState  >> 59))

You may want to test the performance of each, and exercise your profiler....


Writing data to an Excel sheet using openpyxl I am currently writing a program to update and copy data from one spreadsheet to another. The code I have written works fine, but it takes way too long for it to be practical. In total, it takes about an hour to perform this task. The spreadsheets are also very large I might add, one is 20,000 rows by 30 columns, and the other is 3,000 rows by 30 columns. The code updates specific rows in the larger spreadsheet and then copies the data from the smaller spreadsheet onto the larger spreadsheet if that data doesn't already exist there. After analyzing what the problem could be, I found that copying and writing the data to the larger spreadsheet took a majority of the time (~ 55min). The write_only option in openpyxl does not support writing to existing files as I need, so I am stuck as to how to speed up this writing process.

# iterate through ticket column of first sheet
for roww in range (2, sheet.max_row+1):
    sheet1_ticket_number = sheet.cell(row=roww, column = 3).value
    # iterate through ticket column of second sheet
    # Ticket number x from sheet 1 compared to all ticket numbers in sheet 2
    for row2 in range(starting_row, (sheet2.max_row+1+sheet.max_row)):
        sheet2_ticket_number = sheet.cell(row = row2, column = 3).value

        # If ticket number matches, check to see if columns match, if not, update
        if (sheet.cell(row=roww, column = 3).value == sheet2.cell(row = row2, column = 3).value):
            check = 'true'
            for i in range(1, sheet.max_column+1):
                if sheet2.cell(row=row2, column = 3+i).value != sheet.cell(row=roww, column =3+i).value and (3+i != 15) and (3+i != 38) and (3+i != 14):
                    sheet2.cell(row=row2, column = 3+i).value = sheet.cell(row=roww, column =3+i).value
                    #print('updated row# ', row2, 'Column#', 3+i, 'ticket#', sheet2.cell(row=row2, column = 3).value,  'to:', sheet2.cell(row=row2, column = 3+i).value)

                if sheet2.cell(row=row2, column = 1).value is None:
                    sheet2.cell(row=row2, column = 1).value = sheet.cell(row=roww, column =1).value
                if sheet2.cell(row=row2, column = 2).value is None:
                    sheet2.cell(row=row2, column = 1).value = sheet.cell(row=roww, column =1).value
            break

        # if ticket number is not in second file/ empty row, add new ticket row w column entries.
        if (sheet2.cell(row = row2, column = 3).value is None) and (sheet2.cell(row = row2+1, column = 3).value is None):
            sheet2.cell(row=row2, column =3).value = sheet1_ticket_number
            #print('printed new ticket row# ', sheet2.cell(row=row2, column =3).value)
            for j in range(1, sheet.max_column+1):
                if sheet2.cell(row=row2, column = 3+j).value != sheet.cell(row=roww, column =3+j).value:
                    sheet2.cell(row=row2, column = 2).value = sheet.cell(row=roww, column =2).value
                    sheet2.cell(row=row2, column = 1).value = sheet.cell(row=roww, column =1).value
                    sheet2.cell(row=row2, column = 3+j).value = sheet.cell(row=roww, column =3+j).value
            break

Perhaps manipulating data in the spreadsheet is not the most efficient way to do this, could you save the spreadsheet as a csv, and then you could read both spreadsheets into a list structure using the python core csv module.

Alternatively, you might use the Pandas library to import the csv files to a dataframe, do your processing and resave to a csv output.

You can write your results out as csv, open this in excel and resave as an xlxs file.

Both of these approaches should take a few seconds to run


Event Delegation Directive In use cases where web applications utilize ng-repeat heavily and also rely on event triggers for each item in the collection, such as ng-click and the like, watchers become too numerous and may adversely affect performance when the number of items grow large.

To remedy this problem, an event delegation directive could handle the work. It is unfortunate that AngularJS does not support this as a built-in feature, but perhaps they might include it in a future release. Please review my code example for such a directive and point out any areas for improvement or if there's a better implementation.

Note: This example assumes that the view model (not shown) contains an array of items and a function called doSomething.

Update: Here is a working example in JSBin to illustrate its use.

Markup

{{item.name}}

Code

(function () {
  angular.module('myModule').directive('evtDelegate', ['$parse', delegateDirective]);

  function delegateDirective($parse) {
    return {
      restrict: 'A',
      scope: {
        selector: '='
      },
      link: link
    };

    function link(scope, element, attributes) {
      var selector = attributes.evtSelector,
          expression = attributes.evtDelegate,
          expressionHandler = $parse(expression);

      element.on('click', function (e) {
        var targetNode, targetScope;

        targetNode = getTargetNode(element[0], e.target, selector);

        if (targetNode) {
          targetScope = angular.element(targetNode).scope();

          targetScope.$apply(function () {
            expressionHandler(targetScope);
          });
        }
      });
    }

    function getTargetNode(parent, child, selector) {
      if (!parent || !child) {
        return null;
      }

      if (!selector) {
        return child;
      }

      if (parent === child) {
        // we traveresed all the way up to the parent node, so return
        // null since the selector does not exist as a child
        return null;
      }

      // we cannot trust that child.nodeName is uppercase,
      // so we make sure it is before doing the comparison
      if (child.nodeName.toUpperCase() === selector.toUpperCase()) {
        return child;
      }

      return getTargetNode(parent, child.parentNode, selector);
    }
  }
})();

I am puzzled by how you use item both inside and outside ng-repeat.

Each iteration of ng-repeat has its own scope, so its own item. For instance, if items = [item1, item2], which item do you mean to use inside doSomething?
The way it is used now, the item seen by that function will be another item on the outside scope. If this is the intention, this variable should be named differently:

{{item.name}}

UPDATE. Having another look I see better what you are trying to achieve. Still I would improve design by making the directive more re-usable.

You basically want to throw any HTML inside the directive and pick the model associated with event's target. The directive makes this model available on the scope of your expression doSomething(model).  Just like $event made available inside the ng-click. I would then mark this model as something special like:

Then it is the job of the directive to pick the right model and make it available under this name. This way I can change item to my-item inside ng-repeat without breaking things, so the directive is more encapsulated and re-usable.


Given a sequence of positive integers A and an integer T, return true if a continuous sequence of A sums up to exactly T Here is the source of the question, and my solution is below.
Did I determine the worst case correctly?
If you find any input where it doesn't work, please let me know.

public class Main {

    /*
     * http://www.careercup.com/question?id=6305076727513088
     *
     * Worst case: O(2*n)
     * [ 1, 3, 4, 1, 0, 23 ] and k == 23. We have to iterate over the array twice:
     *
     * 1. We sum up all the elements [0, n - 1], n - the array size
     * 2. We subtract all the elements [0, n - 2]
     */
    static boolean findSum(int[] array, int k) {
        int i = 0;
        int start = i;
        int sum = 0;
        for (; i < array.length;) {
            if (array[i] > k) {
                i += 2;
                start = i;
                sum = 0;
                continue;
            }
            if (sum < k) {
                sum += array[i];
                i++;
            }
            while (sum > k) {
                sum -= array[start];
                start++;
            }
            if (sum == k) {
                return true;
            }
        }
        return false;
    }

}

Unit tests.

import static org.junit.Assert.*;

import org.junit.Test;

public class MainTest {

    @Test
    public void testFindSum() {
        // k exists in the input array

        // It's the first element
        assertTrue(Main.findSum(new int[] { 15, 4, 6, 10, 2, 11 }, 15));
        // It's in the middle
        assertTrue(Main.findSum(new int[] { 1, 4, 15, 10, 2, 11 }, 15));
        // It's the last element
        assertTrue(Main.findSum(new int[] { 1, 4, 0, 10, 2, 15 }, 15));

        // k equals to the sum of a few elements

        // The elements are at the beginning
        assertTrue(Main.findSum(new int[] { 10, 4, 4, 10, 2, 16 }, 14));

        // The elements are at the middle
        assertTrue(Main.findSum(new int[] { 1, 4, 4, 10, 2, 16 }, 14));

        // The elements are at the end
        assertTrue(Main.findSum(new int[] { 1, 4, 4, 1, 1, 13 }, 14));

        // k is the sum of all the elements
        assertTrue(Main.findSum(new int[] { 1, 4, 4, 1, 4 }, 14));

        // There is an element which is greater than k
        // It's in the middle
        assertFalse(Main.findSum(new int[] { 1, 42, 4, 1, 4 }, 14));

        // It's the first element
        assertFalse(Main.findSum(new int[] { 42, 1, 4, 1, 4 }, 14));

        // It's the last element
        assertFalse(Main.findSum(new int[] { 1, 0, 4, 1, 42 }, 14));

        // Last but one
        assertFalse(Main.findSum(new int[] { 1, 0, 4, 144, 12 }, 14));

        // Worst case
        assertTrue(Main.findSum(new int[] { 1, 3, 4, 1, 0, 23 }, 23));

    }

}

GitHub

Approach and complexity

It is a pleasure to tell you this: Excellent!

Your approach is precisely what I had in mind when I read the challenge description (before looking at your code).

Unit tests

Excellent that you have unit tests! Definitely helps in checking if it's possible to clean up your method. My only suggestion would be to make them Parameterized, which should be a lot more helpful if one or more of them fails.

Some simplifications

I was slightly confused by this code at first:

if (array[i] > k) {
    i += 2;
    start = i;
    sum = 0;
    continue;
}

Then I noticed that it checked if the value at the current position in the array was more than the target k. Then I figured: "Why use this as a special case? The rest of the code should be able to handle this." And it does. Removing this part of the code doesn't change a thing on your unit tests either. While I can understand why you are doing this (it probably does help a bit with performance in some cases), I don't think it's worth it. Don't add unnecessary special-cases.

Next thing I noticed is that sum < k will always be true. Therefore, that if can be removed and be simply:

sum += array[i];
i++;

And now, speaking of i++, we can transform this for-loop into a traditional for loop, by adding both the initialization and the iteration to it:

for (int i = 0; i < array.length; i++) {

This makes it:

static boolean findSum(final int[] array, final int k) {
    int start = 0;
    int sum = 0;
    for (int i = 0; i < array.length; i++) {
        sum += array[i];
        while (sum > k) {
            sum -= array[start];
            start++;
        }
        if (sum == k) {
            return true;
        }
    }
    return false;
}

Nitpicks

k is a bad variable name. target or targetSum would be better.

I am not a big fan of int start = i; in your original code. start is not dependent on i at all. They start at the same value, sure, but it's more clear to set start = 0;

Your method lacks an accessibility modifier, you probably want to make it a public method.

Overall

Well done!


LockedValue implementation I am working on class for simple sharing values between threads without race-conditions, which can't be a std::atomic (this means, no POD and no trivially-copyable types).
The initial intention was to lock a std::function against writing while reading from an other thread.

The class itself is simple; just lock a mutex at every sensitive part of the class, but I am not totally sure if it's correct or not. I think about to lock the mutex in the dtor, but I don't know if it's good style or not.

#pragma once

template <class T>
class LockedValue
{
private:
    using Lock = std::scoped_lock<std::mutex>;

public:
    LockedValue() = default;

    LockedValue(const LockedValue& _other) :
        m_Value(_other.load())
    {}

    LockedValue(LockedValue& _other) :
        LockedValue(static_cast<const LockedValue&>(_other))
    {}

    LockedValue(LockedValue&& _other) :
        m_Value(_other._takeValue())
    {}

    template <class... Args>
    LockedValue(Args&&... _args) :
        m_Value(std::forward<Args>(_args)...)
    {}

    LockedValue& operator =(const LockedValue& _other)
    {
        auto value = _other.load();
        store(std::move(value));
        return *this;
    }

    LockedValue& operator =(LockedValue&& _other)
    {
        if (this != &_other)
            store(_other._takeValue());
        return *this;
    }

    template <class T_, typename = std::enable_if_t<std::is_convertible_v<T_, T>>>
    LockedValue& operator =(T_&& _value)
    {
        store(std::forward<T_>(_value));
        return *this;
    }

    T operator *() const
    {
        return load();
    }

    template <class T_>
    void store(T_&& _value)
    {
        Lock lock(m_Mutex);
        m_Value = std::forward<T_>(_value);
    }

    T load() const
    {
        Lock lock(m_Mutex);
        return m_Value;
    }

private:
    mutable std::mutex m_Mutex;
    T m_Value;

    T&& _takeValue()
    {
        Lock lock(m_Mutex);
        return std::move(m_Value);
    }
};

EDIT: Is it necessary to check for self-moving in move assign? I think it should be ok to remove the self check.

EDIT2: To the copy ctor related discussion. Delete the non-const& copy ctor from the class and try to use an object like this.

If you don't have c++17 change the Lock alias to std::lock_guard and change std::is_convertible_v to std::is_convertible::value

I pasted it at this online compiler with the ctor already deactivated
click me

// calls the variadic ctor
LockedValue<std::string> s("test");

// this is fine; calls the usual copty ctor
auto u(static_cast<const LockedValue<std::string>&>(s));

/* results in an error because type LockedValue<std::string>&
can not be converted to std::string; this means a second call
for the variadic ctor*/
auto t(s);

You don't need both version of the copy constructor.

    LockedValue(const LockedValue& _other) :
        m_Value(_other.load())
    {}

    LockedValue(LockedValue& _other) :
        LockedValue(static_cast<const LockedValue&>(_other))
    {}

If this is a mutating copy constructor. Then just have the second one. Otherwise have the first one. Since the mutating one simple calls the const version I see no reason for the second one as a non const object will still bind to the const version.

Move operations are usually noexcept. I suppose your are potentially throwing. I would need to look up all the funtions called. You should check and if your move operations are non throwing then you need to make an appropriate comment.

    LockedValue(LockedValue&& _other) :
        m_Value(_other._takeValue())
    {}

A standard move is simply a swap. I suppose you would need a safe swap. Which would be doubly hard as you run into race conditions. But there is always std::lock to help you with that (But its an extra step).

    LockedValue& operator =(LockedValue&& _other)
    {
        if (this != &_other)
            store(_other._takeValue());
        return *this;
    }

Thay's nice. I think I learned a new thing for the day.

    template <class T_, typename = std::enable_if_t<std::is_convertible_v<T_, T>>>
    LockedValue& operator =(T_&& _value)
    {
        store(std::forward<T_>(_value));
        return *this;
    }

And a correct usage of the mutable keyword.

    mutable std::mutex m_Mutex;


optimizing a bash script chain - speed suggestions? i am testing an application that i wrote and want to test the solution my algorithm produces to a Monte carlo solution. I use the harddisk a lot and i was wondering if there was a solution that uses writing data to a file a lot less, since it is really slowing the process down.

The solutions are computed on the nodes of a cluster and examined using this script ( that runs on a node): Parameter $1 is an outputfile that the program wrote.

file=$1
script=/home/hefke/ov_paper/scripts
mv $file.out $file.out.old
grep "Overlapscore:" $file.monte > $file.grepped
awk '/./{print $2}' $file.grepped > $file.overlap
print "$script/std_dev.sh $file.overlap > $file.out"
$script/std_dev.sh $file.overlap > $file.out
cat $file.analy >> $file.out
cat "DONE" >> $file.out

Here is the script that collects the data on the main node. Analy and Monte files are my output files.

echo "Processing outputfiles for the mc_stdev_of_ov"
script=/home/hefke/ov_paper/scripts
curdir=`pwd`
folder=filedata
for file in `ls -1 $curdir/temp_output/$folder/*.analy| sed 's/\(.*\)\..*/\1/'|uniq`
do
    echo $file
    $script/submitter.sh $curdir "processonefile.sh $file.out"
done
echo "$file.out now contains what stdtev spat out."
cat $curdir/temp_output/$folder/*.out >> $curdir/temp_output/tmp.out
awk -f keys.awk $curdir/temp_output/tmp.out >> table.out
cat table.out

How can i optimize this procedure for speed?

It's not related, but please don't mind if I use an "answer" to just comment : it seems I can't comment, maybe because I don't have enough points yet to do so...

Tarrasch, if you still use csh for your shell, please do not script in it.

Please read: http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/

Use instead sh, bash (or even ksh). And better to stick to sh-only because that's what's all unix system rely on (and rc scripts, for example, are based on).


Get signal name from signal number I am trying to get a signal name to print when getting a signal number.
I actually have :
char* getsig(int sig){
    switch (sig)
    {
    case SIGKILL:
        return "SIGKILL";
    case SIGSTOP:
        return "SIGSTOP";
    case SIGTERM:
        return "SIGTERM";
    case SIGTRAP:
        return "SIGTRAP";
    case SIGABRT:
        return "SIGABRT";
    case SIGALRM:
        return "SIGALARM";
    case SIGSEGV:
        return "SIGSEGV";
    case SIGQUIT:
        return "SIGQUIT";
    case SIGINT:
        return "SIGINT";
    case SIGCHLD:
        return "SIGCHLD";
    case SIGCONT:
        return "SIGCONT";
    case SIGPIPE:
        return "SIGPIPE";
    case SIGFPE:
        return "SIGFPE";
    case SIGILL:
        return "SIGILL";
    case 0:
        return "[NO SIGNAL TO DELIVER]";
    default:
        return "UNKN";
    }
}

I can see that this is wrong : I have no certitude that those char* names will be stored in data section on every compiler.
I am certain that there is a clean way of doing such a thing, but I cannot find it on the internet. Do anyone has an idea of how I could do this task the beautiful way?
Edit
The final aim is to print a potential signal delivered by a ptrace(PTRACE_CONT, ...); call, e.g:
printf("ptrace(PTRACE_CONT); [delivered %s]\n", getsig(data));

The way to do this in C is to define an array, and use designated array initializers to ensure the correct mapping between number and string, like so:
static const char *signames[] = {
    [SIGKILL] = "SIGKILL",
    [SIGSTOP] = "SIGSTOP",
    ...
    [0] = "[NO SIGNAL TO DELIVER]",
};

const char *getsig(int sig) {
    if (sig < 0 || sig >= sizeof(signames) / sizeof(*signames) || !signames[sig])
        return "UNKN";

    return signames[sig];
}

Note that it is hard to make this portable; the list of available signals varies by operating system, and some don't even have the concept of signals. Some operating systems and/or standard libraries might already have some function to convert signal numbers to strings, if you target a specific platform and don't care about compatibility, you might consider using that.
Note that strsignal() should be available on any POSIX platform, although it gives you a more human readable string like "Killed" instead of "SIGKILL".


a Python script to format plain text to markdown title I wrote a Python script to format plain text to markdown title.

for example, modify multiple lines

['1 Introduction\n',
 'Part I: Applied Math and Machine Learning Basics\n',
 '2 Linear Algebra\n',
 '3 Probability and Information Theory\n',
 '10 Sequence Modeling: Recurrent and Recursive Nets\n',
 '11 Practical Methodology']

to

['# chapter1: Introduction\n',
 'Part I: Applied Math and Machine Learning Basics\n',
 '# chapter2: Linear Algebra\n',
 '# chapter3: Probability and Information Theory\n',
 '# chapter10: Sequence Modeling: Recurrent and Recursive Nets\n',
 '# chapter11: Practical Methodology']

each chapter title in original file starts with only a number, modify them from 1 to "# chapter 1:", namely, insert "# chapter " before the number, append a colon ":" behind the number; finally write the new toc to a file.

here is the code

import re
# f_name = 'data_multi_lines_3.md'
f_name = 'data_multi_lines.txt'
with open(f_name) as f:
    line_list = f.readlines()
res_list = []
for line in line_list:
    res_list.append(re.sub(r'^(\d{1,2})( +.*?)', r'# chapter\1:\2', line))
with open('your_file.md', 'w') as f:
    for item in res_list:
        f.write("%s" % item)

Is there a better approach to do this?

I guess I should be concerned about the for loop although I have no idea how to improve that.

iterator

There is no need for the intermediate list, just iterate over the text file line per line, and write the parsed line to the output file

variable name

f_name those 3 characters will not make writing or executing the code any slower, but writing it in full can help you understand the code later one

re.compile

You can compile the regular expression

For extra points, you can also name the groups:

re_pattern_chapter = re.compile(r"^(?P<chapter>\d{1,2})(?P<text> +.*?)")
re_pattern_out = r"# chapter(?P=chapter):(?P=text)"

import re

file_name_in = "data_multi_lines.txt"
file_name_out = "your_file.md"

re_pattern_chapter = re.compile(r"^(?P<chapter>\d{1,2})(?P<text> +.*?)")
re_pattern_out = r"# chapter(?P=chapter):(?P=text)"

with open(file_name_in, "r") as file_in, open(file_name_out, "w") as file_out:
    for line in file_name_in:
        line_parsed = re_pattern_chapter.sub(re_pattern_out, line)
        file_out.write(line_parsed)


In your strided filter2, x1 is (1014, 1014, 11, 11).  You are iterating over the 1st 2 dimensions in order to sum on on the last 2.  Let sum do all the work for you, res = x1.sum((2,3))

def filter2(window, x):
    range1 = x.shape[0] - window.shape[0] + 1
    range2 = x.shape[1] - window.shape[1] + 1
    x1 = as_strided(x,((x.shape[0] - 10)/1 ,(x.shape[1] - 10)/1 ,11,11), (x.strides[0]*1,x.strides[1]*1,x.strides[0],x.strides[1])) * window
    res = x1.sum((2,3))
    return res

In my tests this gives a 6x speed improvement.

With numpy iteration, especially nested ones over large dimensions like 1014 is a speed killer.  You want to vectorize this kind of thing as much as possible.

Traditionally Matlab had the same speed problems, but newer versions recognize and compile loops like yours.  That's why your numpy is so much slower.


Avoid re-initialize values in javascript In the below code, variable xAxis is re-initialized multiple times before every push into the array. How can i optimize this code?

var xAxis=1;
            vm.pnLreportingPlotData.push(plotModel.map(function(o){
                    return {x:xAxis++,y:o.PnLreporting};
                })
            );

            var xAxis=1;
            vm.var95PlotData.push(plotModel.map(function(o){
                    return { x:xAxis++, y:o.VaR95Total};
                })
            );

            var xAxis=1;
            vm.var99PlotData.push(plotModel.map(function(o){
                    var xAxis = 1;
                    return  { x:xAxis++, y:o.VaR99Total};
                })
            );

You could remove the need for your xAxis variable through using the index when mapping:

vm.pnLreportingPlotData.push(plotModel.map(function (o, i) {
  return { x: i + 1, y: o.PnLreporting }
}

vm.var95Plotdata.push(plotModel.map(function (o, i) {
  return { x: i + 1, y: o.VaR95Total }
}

vm.var99PlotData.push(plotModel.map(function (o, i) {
  return { x: i + 1, y: o.VaR99Total }
}

Array.prototype.map provides three arguments, currentValue (o), index, and array (which isn't needed in our example)

N.B. you did not need to keep declaring xAxis after you had done so the first time (no need for the multiple var declarations, they could simply become xAxis = 1)

While the above answers your question of how to avoid redeclaring xAxis, I've added a few additional comments below:

Naming

You could possibly benefit from using more descriptive variable names. It is not immediately obvious what variables such as o represent. Likewise, i could be changed to index if you found it helped readability.

Consistency:

var95Plotdata uses lowercase 'd' for 'data', however var99PlotData capitializes it (is this a typo?)
VaR95Total capitalizes both 'V' and 'R' - can this be changed to be consistent with your view model or is this coming from an external api?

Extracting logic

Since the logic is almost identical besides property names, you could extract the logic if you wanted:

function getPlotData(property) {
  // Ideally renaming `o` to whatever it represents
  return plotModel.map(function (o, index) {
    return { x: index + 1, y: o[property] }
  }))
}

vm.pnLreportingPlotData.push(getPlotData('PnLreporting'))
vm.var95Plotdata.push(getPlotData('vaR95Total'))
vm.var99PlotData.push(getPlotData('VaR99Total'))


Using record variant for register mapping in Ada It is OK to model a double usage register with a record variant:

type Capture_Compare_Selection is (Output, Input_TI1, Input_TI2, Input_TRC) with Size=>2;
for Capture_Compare_Selection use(Output=>0, Input_TI1=>1, Input_TI2=>2, Input_TRC=>3);

type TIM_Register_CCMR (CCxS : Capture_Compare_Selection := Output) is record
   case CCxS is
      when Output=>
         OCxFE:Boolean := false;
         OCxPE:Boolean := false;
         OCMx:Output_Compare_Mode := Frozen;
         OCxCE : Boolean := false;
      when Input_TI1 .. Input_TRC =>
         ICxPSC: Input_Prescaler := P_1;
         ICxFF: Timer_Filter := No_Filter;
   end case;
end record with Size=>8;
for TIM_Register_CCMR use record
        CCxS at 0 range 0..1;
        OCxFE at 0 range 2..2;
        OCxPE at 0 range 3..3;
        OCMx at 0 range 4..6;
        OCxCE at 0 range 7..7;
        ICxPSC at 0 range 2..3;
        ICxFF at 0 range 4..7;
end record;

My rationale is that this register (well, fragment of register) has a double usage, depending on the LSB bit; it looks quite like an Ada variant record.

Is there a downside to that or can I commit all the way to this kind of use?

Here is a primer on the documentation:

The full documentation is page 344 of doc RM0090 at ST Microelectronics (huge doc).

I don’t see any problem with your code in itself; this is exactly the sort of thing variant records are for.

I think the only problem with naming record components is that it can lead you into writing code that attempts to directly alter one component, relying on the compiler to do the necessary whole-register reads and writes (the reference document says in para 17.4 that write accesses must be 16- or 32-bits wide, read accesses must be 8-, 16-, or 32-bits wide). There were discussions on comp.lang.ada about the fact that Ada currently has no way to specify this requirement; AI12-0128 refers (for Ada 2020). GNAT now has the pragma/aspect Volatile_Full_Access.

Nevertheless, naming the components will produce much more writable, readable and verifiable code.


Time-dependent state machine I keep having to write state machines that depend on time for various experiments I run and I'd like to know how to write them better. This state machine is for training a neural network by feeding in keys and expected values.

import numpy as np

dt = 0.001
period = 0.1

class SimpleEnv(object):

    def __init__(self, keys, values, env_period=0.1):
        self.keys = keys
        self.values = values
        self.env_idx = np.arange(len(keys))

        self.idx = 0
        self.shuffled = False

        self.i_every = int(round(env_period/dt))
        if self.i_every != env_period/dt:
            raise ValueError("dt (%s) does not divide period (%s)" % (dt, period))

    def get_key(self):
        return self.keys[self.idx]

    def get_val(self):
        return self.values[self.idx]

    def step(self, t):
        i = int(round((t - dt)/dt))  # t starts at dt
        ix = (i/self.i_every) % len(self.keys)
        if ix == 0 and not self.shuffled:
            print("shuffling")
            np.random.shuffle(self.env_idx)
            self.shuffled = True
        elif ix == 1:
            self.shuffled = False
        self.idx = self.env_idx[ix]

        return ix

# note the toy keys and values for testing purposes
s_env = SimpleEnv(np.arange(4), np.arange(1, 5), env_period=period)
key = -1
val = -1
ix = -1
# iterate through keys and values twice
run_time = 4 * period * 2

# the event loop
# starts at dt because of reasons
for t in np.arange(dt, run_time, dt):

    last_ix = ix
    ix = s_env.step(t)
    key = s_env.get_key()
    val = s_env.get_val()
    assert key + 1 == val

    if last_ix != ix:
        print("Key: %s, Value: %s" %(key, val))

The results should look something like:

shuffling
Key: 2, Value: 3
Key: 0, Value: 1
Key: 3, Value: 4
Key: 1, Value: 2
shuffling
Key: 2, Value: 3
Key: 1, Value: 2
Key: 3, Value: 4
Key: 0, Value: 1

How can I write this better or more efficiently? Is there a state machine library in Python that would stop me from having to rewrite variations of this class all the time?

1. (Possible) bugs

There are several problematic aspects of the code in the post. However, it is impossible for me to tell whether these really indicate bugs, because the code has no documentation, and so it is impossible for me to know what it is supposed to do.

The code is not portable to Python 3, because of the use of the true division operator here:

ix = (i/self.i_every) % len(self.keys)

This causes ix to be a float, which results in a warning when evaluating self.env_idx[ix]:

  VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future

The code needs to use the floor division operator, i // self.i_every, to ensure that ix is an integer.
The shuffling logic is fragile: it relies on ix successively taking the value 0 (so that the array is shuffled) and then 1 (so that the shuffled flag is cleared). But this depends on exactly the right sequence of values of t being passed to step. If other values get passed, then the array may never be shuffled:

>>> e = SimpleEnv([1, 2, 3], [10, 20, 30])
>>> for t in np.arange(dt, 100, period * 3):
...     i = e.step(t)
>>>

The output "shuffling" does not appear, indicating that the array was never shuffled.
There is nothing about the interface that stops t from going backwards in time. But if you try to go backwards past a shuffle, then you get different results for the same values of t:

>>> e = SimpleEnv([1, 2, 3], [10, 20, 30])
>>> e.step(period * 1); e.get_key()
(0, 1)
>>> e.step(period * 4), e.get_key()
shuffling
(0, 2)
>>> e.step(period * 1), e.get_key()
(0, 2)

It would be better if the interface were designed so that unsupported actions (like going backwards in time) cannot be attempted.

2. Separation of concerns

An important software design principle is separation of concerns. The idea is that you separate your code into pieces, each of which has a single concern. This makes the pieces easier to understand and easier to test, and makes it more likely that you can reuse some of them.

In this code there are three concerns:

Maintaining corresponding arrays of keys and values.
Generating a sequence of shuffled indexes into the arrays of keys and values, where the indexes are re-shuffled after they have been used.
Converting timestamps into entries in the sequence of shuffled indexes.

Let's see how these might be implemented:

This concern seems very simple. We can just have two variables:

keys = np.arange(4)
values = np.arange(1, 5)

In some circumstances it would make sense to have a state object which varies over time. But we're using NumPy and a rule of thumb when it comes to using NumPy is that we want to do as much as possible with uniform arrays (even if these are big) rather than using iterators that have to evaluate code at each step. So let's write a function that constructs the sequence we need:

def reshuffled_indexes(n, m, state=np.random):
    """Return an array of shape (n,) consisting of shuffled indexes in the
    range [0, m). The indexes are re-shuffled each time they have all
    been used. Optional argument state may be a np.random.RandomState
    instance for random reproducibility.

    """
    out = np.empty((n,), dtype=int)
    r = n % m
    for i in range(0, n - r, m):
        out[i: i + m] = state.permutation(m)
    if r:
        out[-r:] = state.permutation(m)[:r]
    return out

For example:

>>> reshuffled_indexes(17, 4)
array([1, 0, 2, 3, 2, 1, 0, 3, 0, 3, 1, 2, 0, 1, 3, 2, 2])

Note the optional state argument. This is a useful argument to have on any function that produces a random result, because in order to test a function with random results, you need to be able to ensure reproducibility of the random number sequence.
The time series t is converted to an array of indexes by subtracting dt, dividing by the period, and rounding to the nearest integer:

t = np.arange(dt, run_time, dt)
index_t = ((t - dt) / period).round().astype(int)

Now that you have the array of indexes, you know how many reshuffled indexes you are going to need:

r = reshuffled_indexes(index_t[-1] + 1, len(keys))

and you can use this to index the arrays of keys and values to get time series for each:

r_t = r[index_t]
keys_t = keys[r_t]
values_t = values[r_t]


Separating the even and odd numbers in an array in C I wrote a working program in C that takes an array with size 2ⁿ, and separates all the even numbers from the odd numbers.
For example: Input: {1,2,3,4,5,6,7,8}, Output: {8,2,6,4,5,3,7,1}
I want my program to be as efficient as possible and to have space complexity O(log n) and time complexity of O(n log n).
This is the code in C:
#include <stdio.h>

void swap(int*, int*);
void arrange(int arr[], int n);

int main() {
    int arr[] = {1,2,3,4,5,6,7,8};
    int size = sizeof(arr)/sizeof(arr[0]);
    arrange(arr,size);

    return 0;
}

void arrange(int arr[], int n) {

    if (n==1) return;

    int istart = 0, iend = n-1;
    while (istart <= iend) {
    if (arr[istart] % 2 == 0) {
        istart++;
        return arrange(arr+istart,n-1);
    }
    else if (arr[iend] % 2 == 1) {
        iend--;
        return arrange(arr,n-1);
    }
    else {
        swap(&arr[istart++],&arr[iend--]);
        return arrange(arr+istart,n-1);
    }
    }
}

void swap(int *a, int *b) {
    int temp = *a;
    *a = *b;
    *b = temp;
}

How can I improve my code? Maybe make it run faster with a better complexity?

Efficiency
Your code is already more efficient than you think. It has time complexity \$\mathcal{O}(n)\$, and thanks to tail recursion, it also has space complexity \$\mathcal{O}(1)\$.
Avoid forward declarations
I recommend that you avoid the need to forward-declare arrange() and swap() by reversing the order in which the functions appear in your source file. This avoids repeating yourself.
Don't return a void
Unfortunately, the C standard explicitly forbids calling return with an expression in a function returning void, even if the result of that expression itself is also void. While both GCC and Clang allow it, they will produce a warning when the -pedantic option is used. Just separate the call to arrange() from calling the return statement.
Simplify the code
Without making any changes to the algorithm, you can simplify the code. For example, the while-statement is redundant, since its condition will always be true in the first iteration, and you always call return inside the body, so it won't ever repeat. Furthermore, you can avoid having to declare the variables istart and iend. Here is how it could look:
void arrange(int arr[], int n) {
    if (n <= 1)
       return;

    if (arr[0] % 2 == 0) {
        arrange(arr + 1, n - 1);
    } else if (arr[n - 1] % 2 != 0) {
        arrange(arr, n - 1);
    } else {
        swap(&arr[0], &arr[n - 1]);
        arrange(arr + 1, n - 2);
    }
}


Preventing duplicate key errors in EF-Core InMemory provider through locking I have a rest service on Azure AD that calls other rest services. They use AzureAd authentication and AuthContext with a custom TokenCache that stores each users tokens in an InMemory EF Core database. Sometimes I'd get exceptions in AfterAccessNotification when I'd call SaveChanges().

var cache = context.TokenCacheEntries.Find(userObjectId) ?? context.TokenCacheEntries
                                .Add(new TokenCacheEntry { userObjId = userObjectId }).Entity;
                cache.LastWrite = DateTime.Now;
                cache.cacheBits = Serialize();

                // update the DB and the lastwrite
                context.SaveChanges();
                HasStateChanged = false;

A possible cause of this is I have to initialize this context in the Singleton scope because I use it via AuthContext in my JwtBearerEvents.OnTokenValidated method.

I solved the problem with a lock:

    private static object saveLock = new object();

    /// <remarks>
    /// Notification raised after ADAL accessed the cache.
    /// If the HasStateChanged flag is set, ADAL changed the content of the cache
    /// </remarks>>
    private void AfterAccessNotification(TokenCacheNotificationArgs args)
    {

        if (HasStateChanged)
        {
            // SaveChanges() threw an exception that a key already existed once so lets hope this lock fixes that.
            lock (saveLock)
            {
                // retrieve last write from the DB
                var cache = context.TokenCacheEntries.Find(userObjectId) ?? context.TokenCacheEntries
                                .Add(new TokenCacheEntry { userObjId = userObjectId }).Entity;
                cache.LastWrite = DateTime.Now;
                cache.cacheBits = Serialize();

                // update the DB and the lastwrite
                context.SaveChanges();
                HasStateChanged = false;
            }
        }
    }

Is there a better way?

A lock works here, and it's functional, but it does come with some drawbacks in this case.

If you and I both submit the request at the same time, one of them will be delayed even though there wasn't actually a reason. Now you've introduced a potentially hazardous situation, what happens if a lot of distinct persons submit requests at once?

The best option here, is to block only those who violated the key sanity. The easiest way to do this is actually to catch the appropriate exception.

There's a nice question about this on Stack Overflow, which I would like to borrow some information from: Duplicate key exception from Entity Framework?

There is an answer that looks promising, which looks like:

try
{
    await db.SaveChangesAsync();
}
catch (DbUpdateException ex)
{
    SqlException innerException = ex.InnerException.InnerException as SqlException;
    if (innerException != null && (innerException.Number == 2627 || innerException.Number == 2601))
    {
        //your handling stuff
    }
    else
    {
        throw;
    }
}

Based on the research I could suss-out in a short bit, this looks like it's appropriate for EF-Core, whereas the accepted answer looks more like EF on standard. I can't test it, so I leave it to you, but I would go more towards this route than a shared lock.

Of course, it's up to you, but I wouldn't even try to re-save the LastWrite on a duplicate key violation, because we know it's going to be right now, so why do the extra work? I'd just catch and swallow in this case, with a clear comment as to why:

try
{
    context.SaveChanges();
}
catch (DbUpdateException ex)
{
    SqlException innerException = ex.InnerException.InnerException as SqlException;
    if (innerException != null && (innerException.Number == 2627 || innerException.Number == 2601))
    {
        // Swallow the exception, as the only thing we're updating is the `LastWrite`, which will be the current date/time so there's not point in doing additional queries;
    }
    else
    {
        throw;
    }
}

Depending on the logic, I may also even recommend tracking that further up the stack, and only call to this function if it needs updated. (Rather than for every update.)


Why does caching of string objects perform faster? I'm trying to implement String-Pool. I'm maintaining a HashMap<String,String> to store the keys.

public class TestStringCaching {
    static Map < String, String > cache = new HashMap < String, String > ();
    public static void main(String[] args) {
        int iter = 8000000;
        String[] temp = new String[iter];
        long st1 = System.currentTimeMillis();
        for (int i = 0; i < iter; i++) {
            temp[i] = generateString();
        }
        System.out.println(System.currentTimeMillis() - st1);
        st1 = System.currentTimeMillis();
        for (int i = 0; i < iter; i++) {
            temp[i] = generateString1();
        }
        System.out.println(System.currentTimeMillis() - st1);
        System.gc();
    }

    private static String generateString() {
        return new String("abc");
    }

    private static String generateString1() {
        String str = new String("abc");
        if (cache.containsKey(str))
            return cache.get(str);
        else {
            cache.put(str, str);
            return cache.get(str);
        }

    }

}

In both of the cases getString() and getString1() same number of string objects are created. But fetching from the map/cache is blistering fast compared to normal creation of String.

Why is that happening ?

Why skimp on braces?  Putting braces around the if-block in generateString() costs you almost nothing — not even an extra line of code.  Make it a habit to always include explicit braces so that you never write a disaster like this.  (Don't think that it can't happen to you!  Accidents often have more than one contributing factor — so don't be a contributing factor.)

private static String generateString1() {
    String str = new String("abc");
    if (cache.containsKey(str))
        return cache.get(str);
    else {
        cache.put(str, str);
        return cache.get(str);
    }
}

That could be better expressed as:

private static String generateString1() {
    String str = new String("abc");
    if (!cache.containsKey(str)) {
        cache.put(str, str);
    }
    return cache.get(str);
}

If you really don't like braces, putting the action on the same line as the if could also be acceptable:

private static String generateString1() {
    String str = new String("abc");
    if (!cache.containsKey(str)) cache.put(str, str);
    return cache.get(str);
}


Birthday paradox simulation I'm writing a simulator which checks how many balls we need to occupy every box or how many balls we need for any box has at least 2 balls (birthday paradox). I wrote the script in python and it is so slow. My friend wrote a program in C++ and the script is much faster. For boxes_max_num = 1000 it takes few minutes when in my python script it takes 'eternity'.

There is my code:

import numpy as np
import matplotlib.pyplot as plt

def check_every_box_is_occupied(boxes):
    for box in boxes:
        if box == 0:
            return False
    return True

def check_birthday_paradox(boxes):
    for box in boxes:
        if box >= 2:
            return True
    return False

def main():
    number_of_tests = 250
    birthday_paradox_graph = [[], []]
    every_box_is_occupied_graph = [[], []]
    boxes_max_num = 1000
    for number_of_boxes in range(10, boxes_max_num + 1, 1):
        print(number_of_boxes)
        average_frequency_birthday_paradox = 0
        average_frequency_every_box_is_occupied = 0
        for index in range(number_of_tests):
            number_of_balls = 1
            boxes = np.array([0] * number_of_boxes)
            while True:
                boxes[np.random.randint(number_of_boxes)] += 1
                if check_birthday_paradox(boxes):
                    average_frequency_birthday_paradox += number_of_balls
                    break
                number_of_balls += 1
            number_of_balls = number_of_boxes
            boxes = np.array([0] * number_of_boxes)
            while True:
                boxes[np.random.randint(number_of_boxes)] += 1
                if check_every_box_is_occupied(boxes):
                    average_frequency_every_box_is_occupied += number_of_balls
                    break
                number_of_balls += 1

        plt.rcParams.update({'font.size': 15})
        birthday_paradox_graph[0].append(number_of_boxes)
        birthday_paradox_graph[1].append(average_frequency_birthday_paradox / number_of_tests)
        every_box_is_occupied_graph[0].append(number_of_boxes)
        every_box_is_occupied_graph[1].append(average_frequency_every_box_is_occupied / number_of_tests)
    plt.figure(1)
    plt.plot(birthday_paradox_graph[0], birthday_paradox_graph[1], 'ko')
    plt.title("Conajmniej jedna urna ma conajmniej dwie kule")
    plt.xlabel("Liczba urn")
    plt.ylabel("Średnia liczba kul potrzebna do spełnienia warunku")
    plt.figure(2)
    plt.title("Wszystkie urny są zapełnione")
    plt.xlabel("Liczba urn")
    plt.ylabel("Średnia liczba kul potrzebna do spełnienia warunku")
    plt.plot(every_box_is_occupied_graph[0], every_box_is_occupied_graph[1], 'ko')
    plt.show()

if __name__ == '__main__':
    main()

Can you help me to improve the code to make it faster? Is it possible in raw python?

First, let's break some of the loop bodies out into their own functions. If you're having difficulty reasoning about the behavior of your code, then you can definitely afford the (theoretical) performance hit of a function call.

import numpy as np
import matplotlib.pyplot as plt

def check_every_box_is_occupied(boxes):
    for box in boxes:
        if box == 0:
            return False
    return True

def check_birthday_paradox(boxes):
    for box in boxes:
        if box >= 2:
            return True
    return False

def run_test(number_of_boxes):
    number_of_balls = 1
    boxes = np.array([0] * number_of_boxes)
    result = {
        'balls_for_paradox': 0,
        'balls_for_full': 0,
    }
    while True:
        boxes[np.random.randint(number_of_boxes)] += 1
        if check_birthday_paradox(boxes):
            result['balls_for_paradox'] = number_of_balls
            break
        number_of_balls += 1
    number_of_balls = number_of_boxes
    boxes = np.array([0] * number_of_boxes)
    while True:
        boxes[np.random.randint(number_of_boxes)] += 1
        if check_every_box_is_occupied(boxes):
            result['balls_for_full'] = number_of_balls
            break
        number_of_balls += 1
    return result

def run_tests(number_of_boxes, number_of_tests):
    print(number_of_boxes)
    average_frequency_birthday_paradox = 0
    average_frequency_every_box_is_occupied = 0
    for index in range(number_of_tests):
        result = run_test(number_of_boxes)
        average_frequency_birthday_paradox += result['balls_for_paradox']
        average_frequency_every_box_is_occupied += result['balls_for_full']
    plt.rcParams.update({'font.size': 15})
    return {
        'average_frequency_birthday_paradox': average_frequency_birthday_paradox / number_of_tests,
        'average_frequency_every_box_is_occupied': average_frequency_every_box_is_occupied / number_of_tests,
    }

def main():
    number_of_tests = 250
    birthday_paradox_graph = [[], []]
    every_box_is_occupied_graph = [[], []]
    boxes_max_num = 1000
    for number_of_boxes in range(10, boxes_max_num + 1, 1):
        results = run_tests(number_of_boxes, number_of_tests)
        birthday_paradox_graph[0].append(number_of_boxes)
        birthday_paradox_graph[1].append(results['average_frequency_birthday_paradox'])
        every_box_is_occupied_graph[0].append(number_of_boxes)
        every_box_is_occupied_graph[1].append(results['average_frequency_every_box_is_occupied'])
    plt.figure(1)
    plt.plot(birthday_paradox_graph[0], birthday_paradox_graph[1], 'ko')
    plt.title("Conajmniej jedna urna ma conajmniej dwie kule")
    plt.xlabel("Liczba urn")
    plt.ylabel("Średnia liczba kul potrzebna do spełnienia warunku")
    plt.figure(2)
    plt.title("Wszystkie urny są zapełnione")
    plt.xlabel("Liczba urn")
    plt.ylabel("Średnia liczba kul potrzebna do spełnienia warunku")
    plt.plot(every_box_is_occupied_graph[0], every_box_is_occupied_graph[1], 'ko')
    plt.show()

if __name__ == '__main__':
    main()

The above is a pretty literal translation of your code. I introduced some dictionaries as return values; tuples would have worked but would have required some kind of documentation. Explicit classes would be more maintainable, but let's not worry about that for this project.

Next, I'm pretty sure you have a math error in your calculations of the "every box occupied" condition. The line where you reset number of balls should be resetting it to 1, right?
I'm also going to use explicit else statements in this round, because I think they look better.

def run_test(number_of_boxes):
    number_of_balls = 1
    boxes = np.array([0] * number_of_boxes)
    result = {
        'balls_for_paradox': 0,
        'balls_for_full': 0,
    }
    while True:
        boxes[np.random.randint(number_of_boxes)] += 1
        if check_birthday_paradox(boxes):
            result['balls_for_paradox'] = number_of_balls
            break
        else:
            number_of_balls += 1
    number_of_balls = 1
    boxes = np.array([0] * number_of_boxes)
    while True:
        boxes[np.random.randint(number_of_boxes)] += 1
        if check_every_box_is_occupied(boxes):
            result['balls_for_full'] = number_of_balls
            break
        else:
            number_of_balls += 1
    return result

Now we can implement Slothario's suggestion, but we'll have to rethink how we're running these tests. The birthday paradox is easy enough, but to avoid checking every cell for the "all occupied" condition, we need to remember cells we've already visited. We can think of this as crossing items off a list.

def run_test(number_of_boxes):
    number_of_balls = 1
    boxes = np.array([0] * number_of_boxes)
    result = {
        'balls_for_paradox': 0,
        'balls_for_full': 0,
    }
    while True:
        box = np.random.randint(number_of_boxes)
        boxes[box] += 1
        if 2 <= boxes[box]:
            result['balls_for_paradox'] = number_of_balls
            break
        else:
            number_of_balls += 1
    number_of_balls = 1
    boxes = set(range(number_of_boxes))
    while True:
        box = np.random.randint(number_of_boxes)
        if box in boxes:
            boxes.remove(box)
        if not boxes:
            result['balls_for_full'] = number_of_balls
            break
        else:
            number_of_balls += 1
    return result

We have two similar loops. In a lot of contexts I would suggest that they're different enough that they should go in completely separate functions, but since we're going for speed, maybe we should combine them.

def run_test(number_of_boxes):
    number_of_balls = 1
    boxes = np.array([0] * number_of_boxes)
    unoccupied_indexes = set(range(number_of_boxes))
    result = {
        'balls_for_paradox': 0,
        'balls_for_full': 0,
    }
    while not (result['balls_for_paradox'] and result['balls_for_full']):
        box = np.random.randint(number_of_boxes)
        if not result['balls_for_paradox']:
            boxes[box] += 1
            if 2 <= boxes[box]:
                result['balls_for_paradox'] = number_of_balls
        if not result['balls_for_full']:
            if box in unoccupied_indexes:
                unoccupied_indexes.remove(box)
                if not unoccupied_indexes:
                    result['balls_for_full'] = number_of_balls
        number_of_balls += 1
    return result

Let's tidy up some other little things. You had a call to set the font-size of your plot inside one of the loops, I don't think that belonged. I also removed a print statement, which takes up more time than you might think. I'm also replacing several of the loops with list comprehensions, which won't improve performance but it's a little nicer to read. We could go a lot further in this direction, but I wanted to keep the basic structure of your code intact.

import numpy as np
import matplotlib.pyplot as plt

def check_every_box_is_occupied(boxes):
    for box in boxes:
        if box == 0:
            return False
    return True

def check_birthday_paradox(boxes):
    for box in boxes:
        if box >= 2:
            return True
    return False

def run_test(number_of_boxes):
    number_of_balls = 1
    boxes = np.array([0] * number_of_boxes)
    unoccupied_indexes = set(range(number_of_boxes))
    result = {
        'balls_for_paradox': 0,
        'balls_for_full': 0,
    }
    while not (result['balls_for_paradox'] and result['balls_for_full']):
        box = np.random.randint(number_of_boxes)
        if not result['balls_for_paradox']:
            boxes[box] += 1
            if 2 <= boxes[box]:
                result['balls_for_paradox'] = number_of_balls
        if not result['balls_for_full']:
            if box in unoccupied_indexes:
                unoccupied_indexes.remove(box)
                if not unoccupied_indexes:
                    result['balls_for_full'] = number_of_balls
        number_of_balls += 1
    return result

def run_tests(number_of_boxes, number_of_tests):
    results = [run_test(number_of_boxes) for _ in range(number_of_tests)]
    return {
        'average_frequency_birthday_paradox': sum([r['balls_for_paradox'] for r in results]) / number_of_tests,
        'average_frequency_every_box_is_occupied': sum([r['balls_for_full'] for r in results]) / number_of_tests,
    }

def main():
    number_of_tests = 250
    boxes_max_num = 1000
    all_results = [{
                        'n': number_of_boxes,
                        'results': run_tests(number_of_boxes, number_of_tests)
                   }
                   for number_of_boxes
                   in range(10, boxes_max_num + 1, 1)]
    birthday_paradox_graph = [
        [r['n'] for r in all_results],
        [r['results']['average_frequency_birthday_paradox'] for r in all_results]
    ]
    every_box_is_occupied_graph = [
        [r['n'] for r in all_results],
        [r['results']['average_frequency_every_box_is_occupied'] for r in all_results]
    ]
    plt.rcParams.update({'font.size': 15})
    plt.figure(1)
    plt.plot(birthday_paradox_graph[0], birthday_paradox_graph[1], 'ko')
    plt.title("Conajmniej jedna urna ma conajmniej dwie kule")
    plt.xlabel("Liczba urn")
    plt.ylabel("Średnia liczba kul potrzebna do spełnienia warunku")
    plt.figure(2)
    plt.title("Wszystkie urny są zapełnione")
    plt.xlabel("Liczba urn")
    plt.ylabel("Średnia liczba kul potrzebna do spełnienia warunku")
    plt.plot(every_box_is_occupied_graph[0], every_box_is_occupied_graph[1], 'ko')
    plt.show()

if __name__ == '__main__':
    main()

*Runs in ...
my paitence ran out.
But if I follow Pelionrayz and use boxes_max_num=1000 and number_of_tests=1, then it takes about 7 seconds.

(just a heads up that I didn't test ever iteration of this; so i don't know if the earlier versions will actually run.)


Replace subimage by its average value I have the input image in, and the mask image lev (which label each pixel as a number from 1 to nlevels, nlevels=4). I need to get image out, which from each region in==lev will be replaced by its average value.

Here is my brute force approach.  I'm wondering if there is any better way of doing it.

Mat ACA::global_average(Mat in, const Mat &lev)
{
    // in: CV_32F
    // lev: CV_32S
    int M=in.rows;
    int N=in.cols;

    in.convertTo(in, CV_32FC3);

    // vector holds average value for each category
    vector<float> count(nlevels,0);
    vector<Vec3f> average(nlevels);

    int pixlev;
    for (int i=0; i < M; i++)
        for (int j=0; j < N; j++)
        {
            pixlev = lev.at<int>(i,j);
            assert (pixlev<nlevels);

            average[pixlev] += in.at<Vec3f>(i,j);
            count[pixlev]++;
        }

    for (int i=0; i<nlevels; i++)
        average[i] /=count[i];

    Mat out=Mat::zeros(M,N,CV_32FC3);
    for (int i=0; i<M; i++)
        for (int j=0; j<N; j++)
        {
            pixlev = lev.at<int>(i,j);
            out.at<Vec3f>(i,j)=average[pixlev];
        }
    return out;
}

P/S:

Here is an example.

Input Image:

Mask image, it is basically a matrix labelling the pixels into one of 4 categories

Output picture is the average of input image, for each category of pixels. For example, at first I find all pixels has category 1, find their average, and replace all those pixels by the average value.

Algorithm

You will benefit greatly from using OpenCV's built-in functionality rather than performing this operation yourself. OpenCV supports masked operations, which will only apply to pixels where the mask is nonzero.

cv::Mat global_average(const cv::Mat& input, const cv::Mat& levels)
{
    assert(levels.channels() == 1);
    cv::Mat output(input.size(), input.type());
    for (int level = 0; level < nlevels; ++level)
    {
        const cv::Mat mask = levels == level;
        const cv::Scalar avg = cv::mean(input, mask);
        output.setTo(avg, mask);
    }
    return output;
}

Using functionality already present in OpenCV is not only clearer and shorter (by a factor of 3), but also allows the code to be more flexible. You will notice:

There is no explicit type conversion or array indexing occurring, which is safer.
This code is also more flexible: you do not need to convert your images to floating-point or integers.
It can operate on 8-bit images as well, which can give a significant speedup, since most of the time in the algorithm will be spent on memory access. Simply using the raw 8-bit images gives a ~15% speedup in my tests.
It is also possible that OpenCV functions may benefit from hardware and parallelism optimizations, which could make them faster than raw loops.

Code Style

Whitespace

Put spaces around your assignments and comparisons: int i=0; i<m; is harder to read than int i = 0; i < M;. Also, be consistent. Sometimes you do this already, but most of the time you do not.

Braces

You have nested for loops, but only the inner loop has braces. Both loops should have braces surrounding the body for consistency.

Precondition Checking

Although it is good that you leave comments indicating the data types you expect from in and lev, it is better to verify this in code. Adding assertions about these conditions makes it easier to debug if a wrong parameter is passed:

assert(in.depth() == CV_32F);
assert(lev.depth() == CV_32S);

Use const when you can

Both M and N do not change during the function evaluation. They can be declared as const int to allow the compiler to perform checks that they are never accidentally modified. Additionally, you can reduce the scope of pixlev by declaring it inside each of the loops. Then it too can be made const:

const int pixlev = lev.at<int>(i,j); // or use auto to reduce type duplication


Pattern Against Anemic Domain Model I would like to solicit advice on everyone's thoughts on how best to combat the Anemic Domain Model anti-pattern when building out a system based on web services.

One of our goals is to build a set of core web services that expose the most basic services we reuse repeatedly in our organization, which is the creating of domain models.  Right now we have a small library that we share and reuse but as we grow our team it would be much nicer to centralize these basic services.  Over time our systems are going to change as some of the data may come from the cloud (Salesforce.com or AWS) so we're not just isolating basic DAO code in a web service but also application integration.

For example, our customer data comes from the accounting, CRM, and order processing systems.  Configuration is a real pain because every app that ships needs to be bundled with the core library and configuration on each system.  I would like to centralize the creation of models, ala, SOA, but retain a rich model higher up in the Service Layer / Facade.

If you think in general that this is a bad I'd be interested in hearing why!

My thought is to define a domain object Employee that has an EmployeeService injected.  At runtime the EmployeeService implementation is an EmployeeWebServiceClientImpl that implements said interface.  EmployeeWebServiceClientImpl uses a web service proxy to the server.

On the server-side of the web service we have EmployeeWebService invoking EmployeeDao to query the database.  Could just as easily be a class calling out to Salesforce.com to get data.  We would share a library that contained the domain model and interface so you would deserialize the web service response directly into a class that contained the needed business logic.

Below is some example code in order from client to server:

//Example of client
public static void main(String[] args) {
    try {
        Employee employee = Employee.getEmployee("james");
        if (employee.isEligibleForRaise()) {
            System.out.println("Give that man a raise!");
        }
    } catch (RuntimeException e) {
        System.out.println("Oh no!");
    }
}

//Employee Domain Object
public class Employee {

  private String name;

  private String username;

  private static EmployeeService service;

  public static Employee getEmployee(String username) {
    return service.getEmployee(username);
  }

  public static List<Employee> getAllEmployees() {
    return service.getAllEmployees();
  }

  public boolean isEmployeeEligibleForRaise() {
    //business logic here
    return true;
  }

//Getters & Setters

 ...
}

//EmployeeWebServiceClientImpl
public class EmployeeWebServiceClientImpl implements EmployeeService {

  //A client web service proxy to our core basic services
  BaseWebServiceProxy proxy;

  @Override
  public Employee getEmployee(String username) {
    return proxy.getEmployee(username);
  }

  @Override
  public List<Employee> getAllEmployees() {
    return proxy.getAllEmployees();
  }
}

//On the server-side we have EmployeeWebService
public class EmployeeWebService implements EmployeeService {

  EmployeeDao employeeDao;

  @Override
  public List<Employee> getAllEmployees() {
    return employeeDao.getAllEmployees();
  }

  @Override
  public Employee getEmployee(String username) {
    return employeeDao.getEmployee(username);
  }
}

Is that making sense?  Basically the plan is to keep core business logic in the Employee domain object but isolate the data access logic in a web service.

Thoughts?

You're adding a lot of complexity here. Plus, you're going against the point of a Service Layer and Domain Model. Your Domain Model should probably use either Active Record or Data Mapper.

The point of a Service Layer is to act as an end point (API) that holds common business logic that deals with integration of domain model objects. Your service layer should just query repositories and delegate calls to process business logic to your domain model (which should hold as much business logic as possible). Injecting it into your domain model adds persistence concerns, which it should not care about.


Merging and sorting overlapped intervals We are given an array of intervals we need to merge all overlapping intervals and sort the resulting non-overlapping intervals. When they "touch" in a single point, intervals are also considered to be overlapping .
class Span
{
    public uint Start { get; }
    public uint End { get; }

    private Span(uint start, uint end)
    {
        Start = start;
        End = end;
    }

    private static Span _default = new Span(0, 0);

    // To avoid constructor throwing an exception
    public static Span Create(uint start, uint end)
    {
        if (start > end) throw new ArgumentOutOfRangeException(nameof(start), "Begin cannot me more than end");
        return new Span(start, end);
    }

    public bool IsOverlapped(Span other)
    {
        return Start <= other.End && End >= other.Start;
    }

    public Span Merge(Span other)
    {
        if (!IsOverlapped(other)) throw new ArgumentOutOfRangeException(nameof(other), "Spans must overlap");
        return new Span(Math.Min(Start, other.Start), Math.Max(End, other.End));
    }

    public bool TryMerge(Span other, out Span mergedItem)
    {
        mergedItem = _default;
        if (!IsOverlapped(other)) return false;
        mergedItem = new Span(Math.Min(Start, other.Start), Math.Max(End, other.End));
        return true;
    }
}

// In reality this method will be in a different class where it belongs
class Util
{
    public static Span[] Normalise(Span[] spans)
    {
        var results = new List<Span>();
        foreach (var item in spans.OrderBy(x => x.Start))
        {
            var lastResultIndex = results.Count - 1;
            if (lastResultIndex >= 0 && results[lastResultIndex].TryMerge(item, out Span mergedItem))
            {
                results[lastResultIndex] = mergedItem;
            }
            else
            {
                results.Add(item);
            }
        }
        return results.ToArray();
    }
}

I can't see the point in having the Create method here, there's no reason why a constructor can't throw an exception in C#. I'd just get rid of Create and move it all into the constructor (making it public).
The name IsOverlapped sounds a bit odd to me. Is is present tense and Overlapped is past tense. Maybe something like IsOverlapping or even just Overlaps.
You have a typo here: "Begin cannot me more than end". me should be be.
If you have access to expression-bodied members, they can reduce noise IMO:
public bool Overlaps(Span other) => other != null && Start <= other.End && End >= other.Start;

You'll notice that I've also put a null check for other in there at the same time.
You should rewrite Merge to call TryMerge internally:
public Span Merge(Span other) =>
    TryMerge(other, out var mergedSpan)
        ? mergedSpan
        : throw new ArgumentOutOfRangeException(nameof(other), "Spans must overlap");

I really value conciseness and I'm happy with throw expressions so I like this kind of pattern. If you don't, this version would also be fine:
public Span Merge(Span other)
{
    if (!TryMerge(other, out var mergedSpan))
        throw new ArgumentOutOfRangeException(nameof(other), "Spans must overlap");
    return mergedSpan;
}

Note: there's no need to check for null here as we now do that in Overlaps.
Your Normalise method looks good to me. It's a simple algorithm and easy to read.


A tool to extract all links from a web-page in Python I'm starting writing in Python and is tasked with writing a simple tool that opens a web-page, collects all the URLs (both complete like http://mail.com and relative ones like /mail/inbox) and returns a list of them prefixed with the domain name. The code is below.

Could someone take a look and advise on what could've been better in terms of the code style, readability, general coding principles and code quality?

Appreciate your help!

import sys, requests
from bs4 import BeautifulSoup
from urllib.parse import urlparse

def main(argv):
    ''' Main '''
    links = []

    try:
        if len(argv) != 3:
            raise IndexError("Not enough arguments.")

        url = argv[2].rstrip('/')
        scheme = urlparse(url).scheme

        if argv[1].upper() in ['GET', 'POST', 'OPTIONS', 'PUT', 'DELETE']:
            method = argv[1]
        else:
            raise ValueError("Method %s is not supported." % argv[1])

        r = requests.request(method, url, verify=False) # get the response
        url = r.url.rstrip('/') # if there was a redirect - acquire the URL from the response

        print('[+] Retrieving '+url)
        raw_html = r.content

        soup = BeautifulSoup(raw_html, "html.parser")

        # neat print headers
        print("[+] Received the response HTTP %d" % r.status_code)
        for header in r.headers:
            print(header + ':', r.headers[header])

        # gathering a list of hrefs and srcs
        for script in soup.find_all('script'):
            if not script == None:
                links.append(script.get('src'))
        for anchor in soup.find_all('a'):
            if not anchor == None:
                links.append(anchor.get('href'))
        for link in soup.find_all('link'):
            if not link == None:
                links.append(link.get('href'))
        for form in soup.find_all('form'):
            if not form == None:
                links.append(form.get('action'))

        links = list(filter(None, links)) # removing None src's

        links = [link for link in links if
                 not link[0] == '#' and
                 not urlparse(link).scheme in ['mailto', 'skype'] and
                 not link == '/'] # removing bookmarks, emails and skype and '/'

        links = [scheme + '://' + link[2:] if link[0:2] == '//' else
                 link if urlparse(link).scheme in ['http', 'https'] else
                 url + link if link[0] == '/' else
                 url+'/'+link for link in links] # appending URL value

    except Exception as e:
        print("[-] Something went wrong: %s" % e)

    except KeyboardInterrupt as e:
        print("[x] Exiting by user command")

    print("\n[+] %d Links extracted:" % len(links))
    for link in links:
        print(link)
    return links

if __name__ == "__main__":
    print('[*] Starting the main module')
    main(sys.argv)

Code style issues (from top to bottom)

imports - you should properly group imports as per PEP8 - in the first group you would have your system-level imports, then third-party libraries and then your local imports - all groups sorted alphabetically with a blank line between the groups:

import sys
from urllib.parse import urlparse

from bs4 import BeautifulSoup
import requests

docstrings should be enclosed into triple quotes, start with a capital letter and end with a dot. "Main" is though not a descriptive and useful documentation string - see if you can add details about what the function does
variable naming - use descriptive variable names - response instead of r
watch for spaces around the operators
when comparing to None, you should be using is and not ==, e.g. if anchor is not None - you though actually don't need these comparisons - see below
avoid using bare exception clauses

Other improvements

you should switch to argparse module instead of manually parsing and validating sys.argv
you don't actually need checks against None and filtering the links list - you can enforce the presence of attributes this way:

script_elements = [element['src'] for element in soup.select('script[src]')]
anchor_elements = [element['href'] for element in soup.select('a[href]')]
link_elements = [element['href'] for element in soup.select('link[href]')]
form_elements = [element['action'] for element in soup.select('form[action]')]

links = script_elements + anchor_elements + link_elements + form_elements

if performance is critical, see if you can use lxml instead of html.parser (requires lxml module to be installed):

soup = BeautifulSoup(response.content, "lxml")

see if you can join the current url with the relative link with urljoin() - might simplify the post-processing for the extracted links


Cor.test inside for loop is very slow for large matrix processing I have a matrix of  36,000 genes measurements for 60 samples. For calculating most correlated genes of the first gene in the matrix, my following program gives the result. However processing time is extremely slow.

dat <- as.matrix(read.table("/path/gene_matrix.txt", header = TRUE, fill = TRUE))
corr_list <- data.frame(top=numeric(), correlated=numeric(), cor=numeric(), p.value=numeric())

for (i in 2:nrow(dat)) {
r <- cor.test(dat[1,], dat[i,])
corr_list[i-1,] <- c(rownames(dat)[1], rownames(dat)[i], r$estimate, r$p.value)
}

corr_list <- corr_list[order(corr_list$cor, decreasing = TRUE), ]

write.table(corr_list, "/path/most_related.txt", quote = FALSE, row.names = FALSE, sep="\t")

Anyone could suggest an efficient method for the above problem.

If you had profiled your code, you would have seen that unusually long time is taken by line:
corr_list[i - 1,] <- c(rNames[1], rNames[i], r$estimate, r$p.value)
It is probably because you are combining multiple type values into one vector.
So this should be mush faster:
f3 <- function(dat) {
  require(data.table)
  mainVar <- dat[1,]
  rNames <- rownames(dat)
  rez <- lapply(2:nrow(dat), function(i) {
    r <- cor.test(x = mainVar, y = dat[i,])
    c(r$estimate, r$p.value)
  })
  fin <- data.table(rNames[1], rNames[-1])
  fin <- cbind(fin, data.table(do.call(rbind, rez)))
  setnames(fin, c("top", "correlated", "cor", "p.value"))
  return(rez)
}

I <- 60
N <- 1000
dat <- MASS::mvrnorm(N, mu = rep(0, I), diag(I))
rownames(dat) <- paste0("G", 1:N)

# Comparison
res1 <- microbenchmark(
  f1(dat),
  f2(dat),
  f3(dat),
  times = 10
)
print(res1, unit = "s")
# Unit: seconds
# expr       min        lq      mean    median        uq       max neval cld
# f1(dat) 0.4333447 0.4609346 0.4880110 0.4863479 0.5050751 0.5782526    20   c
# f2(dat) 0.3585121 0.3729532 0.4021378 0.3979831 0.4269367 0.4899676    20  b
# f3(dat) 0.1483598 0.1610634 0.1782496 0.1787396 0.1934256 0.2273650    20 a

autoplot(res1)

Update
If we calculate correlation with base function and p.value - ourselves, we can get rid of for loop and do all of this much faster. Only problems my arise if your data contains missing values:
pcor <- function(r, n){
  t <- r * sqrt(n - 2) / sqrt(1 - r^2)
  p <- 2 * (1 - pt(abs(t), (n - 2)))
  p[p > 1] <- 1
  p
}

f4 <- function(dat){
  require(data.table)

  rNames <- rownames(dat)
  d2 <- t(dat)
  cors <- cor(d2[, 1], d2[, -1])
  cors <- t(cors)
  cp <- pcor(cors, ncol(dat))
  cp <- cbind(cors, cp)
  fin <- data.table(rNames[1], rNames[-1])
  fin <- cbind(fin, cp)
  setnames(fin, c("top", "correlated", "cor", "p.value"))
  return(fin)
}

I <- 60
N <- 1000
dat <- MASS::mvrnorm(N, mu = rep(0, I), diag(I))
rownames(dat) <- paste0("G", 1:N)

res1 <- microbenchmark(
  f1(dat),
  f2(dat),
  f3(dat),
  f4(dat),
  times = 10
)
print(res1, unit = "s")
# Unit: seconds
# expr         min          lq        mean      median          uq         max neval  cld
# f1(dat) 0.408167816 0.410614574 0.428363371 0.429430744 0.442042920 0.456887399    10    d
# f2(dat) 0.334446197 0.345779175 0.366422216 0.362465218 0.378226388 0.426838502    10   c
# f3(dat) 0.139088268 0.145095289 0.156048277 0.153321298 0.162591518 0.183544069    10  b
# f4(dat) 0.002363351 0.002428473 0.002615854 0.002483812 0.002797716 0.003117556    10 a


Binary to decimal (and back) converter in c++ I'm mainly looking for error checking / type safety advice.
main.cpp
#include <iostream>
#include <string>
#include <cmath>
#include <numeric>

void binaryToDecimal();
void decimalToBinary();

int main()
{
    std::ios::sync_with_stdio(false);

    bool running = true;
    int ans;

    while (running)
    {
        std::cout << "1. Binary to decimal\n2. Decimal to binary\n3. Quit\n";
        std::cin >> ans;

        std::cout << '\n';

        switch (ans)
        {
        case 1:
            binaryToDecimal();
            break;

        case 2:
            decimalToBinary();
            break;

        case 3:
            running = false;
            break;

        default:
            break;
        }
    }

    return 0;
}

void binaryToDecimal()
{
    int out = 0;
    std::string in;

    std::cout << "Input binary number: ";
    std::cin >> in;

    int i = in.size() - 1;
    if (i > 32)
    {
        std::cerr << "That number is too big!\n\n";
        return;
    }

    for (int j = 0; i > -1; --i, ++j)
    {
        out += static_cast<int>(std::pow(2, i)) * (in[j] - '0');
    }

    std::cout << in << " when converted to decimal is " << out << "\n\n";
}

void decimalToBinary()
{
    std::string out = "";
    std::string in;
    std::cout << "Input decimal number: ";
    std::cin >> in;

    if (in.size() > std::to_string(std::numeric_limits<int>::max()).size())
    {
        std::cerr << "That number is too big!\n\n";
        return;
    }

    int mod = std::stoi(in);

    while (mod > 0)
    {
        out += (mod % 2) + '0';
        mod /= 2;
    }
    std::reverse(out.begin(), out.end());

    std::cout << in << " when converted to binary is " << out << "\n\n";
}

General Observations

I'm mainly looking for error checking / type safety advice.

There doesn't seem to be a lot of error checking on user input and user input is one of the places where error checking is most important. For instance, the code doesn't handle negative numbers but there is not checking for negative input.
The code also doesn't check for lack of input, and there are no error messages when the input is wrong.
Type Safety
Since the program doesn't process negative numbers it might be better to use unsigned or unsigned int for conversion rather than int.
Unnecessary Function Prototypes
If the main() function were placed at the bottom of the file rather than the top of the file, the function prototypes
void binaryToDecimal();
void decimalToBinary();

are not needed.
Magic Numbers
There is a Magic Numbers in the binaryToDecimal() function (32), it might be better to create symbolic constants for them to make the code more readable and easier to maintain. These numbers may be used in many places and being able to change them by editing only one line makes maintenance easier.
Numeric constants in code are sometimes referred to as Magic Numbers, because there is no obvious meaning for them. There is a discussion of this on stackoverflow.
In this case 32 probably refers to word size, but that is an arbitrary number, most computers these days have a 64 bit word size. You could get the actual size of the word by using the sizeof() function and then multiplying it by 8 since sizeof() the number of char in the object and 8 bits is the size of a char, there also might be a word size constant hidden in an include header.
Complexity
While main() is pretty simple now, once you start adding the error checking you need it will get more complex, and it should be broken up into different function.


If you want not to have callback hell, my suggestion is to use classes. You can find a lot of techniques for how to define a class. I did it my way. You may change what you want. I know that many people would suggest some changes, but that will be about specific rules, not about the full decision. I suppose a majority would agree that my way has a right to live.

My problem while developing a class was that I did not know about the  semantics of your data. It is a very good start to find out the semantics, split by items, give them names. Then you would get a decision from your logic. I gave some names to variables and methods but I suppose it likely should be changed.

function Transfer(source) {
    this.source = source;
    this.result = {};
}

Transfer.prototype = {
    constructor: Transfer,

    make: function() {
        var self = this;
        _.each(this.source, function (val, key) {
            self.processNames(val, key);
        });
        return this.result;
    },

    processNames: function(data, name) {
        var self = this;
        var path = [name];
        _.each(data, function (val, key) {
            self.processIndexes(val, path);
        });
    },

    processIndexes: function(data, path) {
        var self = this;
        _.each(data, function (val, key) {
            var newPath = (path).slice(0);
            newPath.unshift(key);
            self.processCodes(val, newPath);
        });
    },

    processCodes: function(data, path) {
        var current = this.result[path[0]] = this.result[path[0]] || {};
        current = current[path[1]] = current[path[1]] || [];
        current.push(data);
    }
};

var transfer = new Transfer(initial_obj);
transfer.make();

One more improvement might be to add a static method, e.g. .make:

Transfer.make = function (source) {
    var inst = new this(source);
    return inst.make();
};

I would not advise you to return any values from a constructor, as it is just for setting options. Generally, I would use controllers only for defining properties. Otherwise you need a stub prototype while testing which is not an elegant design.

I tested your code. It works as expected. I did not use indexes when setting values to array. I just used Array#push. If you have the structure that you provided, it should work.

I know that this technique has more lines of code than yours. But it has several advantages:

it is open for testing
it is open for extending
it is more human readable
it is more clear


PHP backup log table from MySQL I have been working on a little project to tidy up my log table which has exceeded the shared host provider limits:) So I have done some script that is working except one thing at the end: optimise DB.

This script will run in cron so I hope security should not be issue.

My questions are:

the code is working, however I would like to learn how to do it better, so If you could point out thing that I shouldn't do and I did I would appreciate
if someone could tell me why my optimise DB doesn't work I would also thanks.
Performance!!! It will run on a table which has 1million rows and I am worried that his script wont perform fast enough. Any idea how to work this out?

I know I should use PDO (next step - as I am not familiar with it yet) but as a first step I am proud that I could put together something that is working.

<?php
include 'opendb.php';

//functions
function test($string){
  echo "<p>".$string."</p>";
}

function db_rows($db,$ord){
  $dbquery="SELECT azon FROM $db ORDER BY azon $ord LIMIT 1";
  $dbresult=mysql_query($dbquery);
  $row = mysql_fetch_array($dbresult);
  $dbrow = $row['azon'];
  return $dbrow;
}
// end of functions

//config information...
$acttable  = 'foo';
$today = date("yW_Hi");
$newdb = 'test_'.$today;

$firstact= db_rows($acttable,"asc");
$lastact= db_rows($acttable,"desc");
$upto=$firstact+25000;

test($lastact);
test($firstact);

if ($lastact-$firstact>50000) {

//create a new table
$newdbsql=" CREATE TABLE  $newdb ( `azon` bigint(20) NOT NULL AUTO_INCREMENT,  `mikor` datetime NOT NULL,  `felhazon` int(11) NOT NULL,  `felhnev` varchar(255) NOT NULL,  `muvelet` varchar(20) NOT NULL,  `sql` varchar(255) NOT NULL,  `tabla` varchar(255) NOT NULL,  `mezok` varchar(255) NOT NULL,  `ertekek` text NOT NULL,  `feltetel` varchar(255) NOT NULL,  `ip` varchar(255) NOT NULL,  `bongeszo` varchar(255) NOT NULL,  PRIMARY KEY (`azon`),  KEY `mikor` (`mikor`),  KEY `felhazon` (`felhazon`),  KEY `felhnev` (`felhnev`),   KEY `muvelet` (`muvelet`),  KEY `tabla` (`tabla`),  KEY `feltetel` (`feltetel`)) ENGINE=MyISAM  DEFAULT CHARSET=utf8 AUTO_INCREMENT=3524772 ;" ;
test($newresult);
$newresult = mysql_query($newdbsql);

// copy all the data
$query = "INSERT INTO $newdb SELECT * FROM $acttable WHERE $acttable.azon < $upto";
test($query);
$result = mysql_query($query);

// so what has happened...
if ($result) {
   test("ok");
  $delquery = "DELETE FROM $acttable WHERE $acttable.azon < $upto";
  test($delquery);
  $delresult = mysql_query($delquery);
  if ($delresult) {
     test("Deleted rows - OK");
  } else {
     test("failed to delete...");
     // then tidy up everything:)
    $res = mysql_query('SHOW TABLE STATUS WHERE Data_free / Data_length > 0.1 AND Data_free >  102400');
    while($optrow = mysql_fetch_assoc($res)) {
             mysql_query('OPTIMIZE TABLE ' . $optrow['Name']);
         }
      }
   }
   else {
      test("failed to copy...");
   }

}
else {
  test("no work needs to be done.");
}

// close db
mysql_close($conn);
?>

Your Select statement.

SELECT azon FROM $db ORDER BY azon ASC/DESC LIMIT 1

you could simplify the statement like this:

SELECT MAX(azon)/MIN(azon) AS 'azon' FROM $db

There are some discussions about the gain of performance by using MIN/MAX instead of ORDER BY + LIMIT. I personally experienced that MIN/MAX is faster, especially on tables with many rows.
You can simply copy a table structure by using

CREATE TABLE `$newdb` LIKE `$acttable`

but you'd have to set the AUTO_INCREMENT manually after this, otherwise it would be set to 1.

ALTER TABLE `$newdb` AUTO_INCREMENT=3524772

(I don't see the purpose on this, maybe it's not even nessecary?)
Your optimize.

I'm quite irritated about the way to want to query this. You're calling that statement inside the else-condition -> after your delete has failed. It should be called inside the if-condition (or I missunderstood the goal).
You should consider using InnoDB instead of MyISAM (in general), because there would be no need (and possibility) of running an OPTIMIZE.

There should be no performance issue on the php-side as long as you do not iterate through the many rows.
You may want to increase the max_execution_time-directive to prevent the script from stopping, but I consider this is not nessecary.

On MySQL-side, i really suggest changing to InnoDB (you may want to read this).

Conclusion:
Besides the little changes of the statements and the use of OPTIMIZE inside your if-condition, there are no required improvements for exactly this script in my opinion. I strongly recommend to change the database engine when you make your changes for using PDO. This of course means more effort, as there are many variables which you can or should set to the InnoDB engine, but I think it is worth it. In fact, there aren't real advantages on using MyISAM, less then ever in performance.


Syracuse conjecture in OCaml I am wondering how to optimize the following OCaml simple code.

#load "unix.cma";;
open Unix;;

let time f x =
    let start = Unix.gettimeofday ()
    in let res = f x
    in let stop = Unix.gettimeofday ()
    in let () = Printf.printf "Execution time: %fs\n%!" (stop -. start)
    in res ;;

let syracuse x =
    let rec aux x t =
        match x with
          1 -> t
          |x -> if x mod 2 = 0 then aux (x / 2) (t + 1) else aux (3 * x + 1) (t+1) in
     aux x 0 ;;

let syracuse2 x =
    let rec aux x t =
        if x = 1 then
            t
        else
            if x mod 2 = 0 then aux (x / 2) (t + 1) else aux (3 * x + 1) (t+1)
        in
     aux x 0 ;;

let rec iterate_over n f l  =
     if n = 0 then l else iterate_over (n-1) f ((f n)::l);;

let syracuses n = iterate_over n syracuse [];;
let syracuses2 n = iterate_over n syracuse2 [];;

(time syracuses 100000);;
(time syracuses2 100000);;

I made a first attempt replacing the match with with ifs but it slightly reduced the performance.

Execution time: 1.880309s
Execution time: 1.920133s

I tried to replace the t+1s by succ t but it did not bring anything. I think that there are some redundancies in the operations performed when calling mod 2 and dividing by 2 but I don't know how it can help.

Okay, so let's forget about syracuses2 because using ifs is indeed slower and let's stop depending on Unix which makes things less portable and slightly more difficult to compile.

let time f x =
    let t = Sys.time() in
    let fx = f x in
    Printf.printf "Execution time: %fs\n" (Sys.time() -. t);
    fx

let syracuse x =
    let rec aux x t =
        match x with
            | 1 -> t
            | x ->
                if x mod 2 = 0 then
                    aux (x / 2) (t + 1)
                else
                    aux (3 * x + 1) (t+1) in
     aux x 0

let rec iterate_over n f l =
     if n = 0 then l else iterate_over (n-1) f ((f n)::l)

let syracuses n = iterate_over n syracuse [] ;;

time syracuses 100000 ;;

This takes 0.4s on my machine and 0.03s when compiled (ocamlopt syracuse.ml && ./a.out).

Keep in mind that integer arithmetic is about twice as slow as in C because integers in OCaml are tagged. Nothing we can do about this.

syracuse looks fine by itself but iterate_over is highly inefficient because you're computing the same values over and over again. Once you know that a given integer n ends up with a 1, you can remember this. This is called memoization:

open Printf

let time f x =
    let t = Sys.time() in
    let fx = f x in
    Printf.printf "Execution time: %fs\n" (Sys.time() -. t);
    fx

let syracuse n =
    let rec aux x t cache =
        if x < n && Array.get cache x <> 0 then
            Array.get cache x
        else
            match x with
                | 1 -> t
                | x ->
                    if x mod 2 = 0 then
                        aux (x / 2) (t + 1) cache
                    else
                        aux (3 * x + 1) (t + 1) cache in
    let cache = Array.init n (fun x -> 0) in
    for i = 1 to n-1 do
        Array.set cache i (aux i 0 cache)
    done ;;

time syracuse 100000 ;;

This takes 0.035s on my machine and 0.003s when compiled: a 10x speedup! This is a space-time tradeoff. Another optimization allows you get a k-speedup with a 2^k precomputation, but it might not play well with the cache.

Now, to get maximum performance, you probably want to get down to C, given the relative slowness of integer arithmetic in OCaml.