IP Scanner via python sockets Please review my code for scanning active IP address . Please Suggest improvements ...

import socket
import threading
import sys

ip0=input('STARTING IP  : ')
ip1=input('ENDING IP    : ')
port=int(input('PORT NUMBER  : '))
timeout=int(input('TIMEOUT      :'))
timeout=timeout/1000
print()
ip0=list(map(int,ip0.split(".")))
ip01=list(map(int,ip1.split(".")))
class IpAddr:
    def __init__(self,d,c,b,a):
        self.a=a
        self.b=b
        self.c=c
        self.d=d
    def increase(self):
        self.a+=1
        if self.a>255:
            self.a=0
            self.b+=1
        if self.b>255:
            self.b=0
            self.c+=1
        if self.c>255:
            self.c=0
            self.d+=1
        return str(self.d)+"."+str(self.c)+"."+str(self.b)+"."+str(self.a)
    def data(self):
        return str(self.d)+"."+str(self.c)+"."+str(self.b)+"."+str(self.a)
    def next(self):
        self.b+=1
        if self.b>255:
            self.b=0
            self.c+=1
        if self.c>255:
            self.c=0
            self.d+=1
        return str(self.d)+"."+str(self.c)+"."+str(self.b)+"."+str(self.a)
    def diff(self,x,y,z,w):
        return (x-self.d)*256**2+(y-self.c)*256+z-self.b

def find(z,i):
    i=list(map(int,i.split(".")))
    ip=IpAddr(i[0],i[1],i[2],i[3])
    global cnt
    while i!=z:
        i=ip.data()
        s=socket.socket()
        s.settimeout(timeout)
        try:
            s.connect((i,port))
            print('[*] ACTIVE HOST',i)
            cnt+=1
        except:
            pass
        s.close()
        i=ip.increase()
ip=IpAddr(ip0[0],ip0[1],ip0[2],ip0[3])
n=ip.diff(ip01[0],ip01[1],ip01[2],ip01[3])-1
cnt=0
d={}
for i in range(n):
    p,q=ip.data(),ip.next()
    d[i]=threading.Thread(target=lambda:find(q,p))
    d[i].daemon=True
    d[i].start()
find(ip1,ip.data())

Starting with Python 3.3, there is a standard ipaddress module.  Avoid reinventing-the-wheel.

Your IpAddr class is cumbersome because you store the four dotted-quad components separately.  It would be easier to represent it internally as a 32-bit integer, and convert it from/to dotted-quad notation in the constructor and the __str__ method.  For those conversions, you may find struct.pack() and struct.unpack() useful.

I would expect IpAddr objects to be immutable; the .next() method should return a new object instead of mutating the original one.

When scanning a range of IP addresses, watch out for network and broadcast addresses.  Opening a connection on a broadcast address would be inappropriate.

The find function is poorly named, and its parameters z and i are cryptic.  In fact, your variable names are generally horrible — what do p, q, n, and d mean?

Using a global cnt variable is inappropriate — especially since you access it in threads without any mutex.  But why even bother, if it's just a write-only variable whose value never matters?


Producer-Consumer using low level synchronization I saw a tutorial from a site where it showed the use of ArrayBlockingQueue as a thread-safe concurrent data-structure . Just for learning purposes , i tried to build something similar using synchronized and wait-notify methods in java. I'm really very new to java.util.concurrent and i'm hoping someone could guide me to some improvements and point out the bad parts of this code i wrote :

import java.util.ArrayList;
import java.util.Random;

public class LowLevelProducerConsumer {
    ArrayList<Integer> list = new ArrayList<Integer>();
    private final int size = 10;
    private int elems = 0;
    Object lock = new Object();
    Random r = new Random(System.currentTimeMillis());

    public void producer() {
        while (true) {
            synchronized (lock) {
                try {
                    while (elems != size) {
                        list.add(r.nextInt(100));
                        elems++;
                    }
                } finally {
                    // allows consumer to remove an entry
                    lock.notify();
                }
            }
        }
    }

    public void consumer() throws InterruptedException {
        Thread.sleep(100); // any better way to ensure producer runs first ?
        int ran = 0;
        while (true) {
            // consumer tries to acquire lock here , but it can do so only after
            // producer has called notify
            synchronized (lock) {

                // do i need a lock.wait() here somewhere ?

                ran = r.nextInt(10);

                if (ran == 7) { // just an arbitrary condition

                    int loc = r.nextInt(list.size());
                    int data = list.remove(loc);
                    System.out.format(
                            "%d removed from index %d , size : %d\n",
                            data, loc, list.size());
                    // decrementing elems to let the producer work again
                    elems--;
                    Thread.sleep(300);
                }

                // release the lock so that producer can fill things up again
                lock.notify();
            }
        }

    }

    public static void main(String[] args) {
        final LowLevelProducerConsumer low = new LowLevelProducerConsumer();
        Thread t1 = new Thread(new Runnable() {
            @Override
            public void run() {
                low.producer();
            }
        });
        Thread t2 = new Thread(new Runnable() {
            @Override
            public void run() {
                try {
                    low.consumer();
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        });

        t1.start();
        t2.start();
        try {
            t1.join();
            t2.join();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

    }
}

ps : i'v kept my understanding of how the code works in the comments. Any correction to them would be much appreciated.

Miscellaneous Remarks

Your elems variable is just list.size(), so why not just use list.size() everywhere and eliminate elems?
Your other size is a constant, and should be called SIZE by convention.
Instead of spreading out int ran = 0; ran = r.nextInt(10);, and if (ran == 7) { ... } over three lines, you should just do if (r.nextInt(10) == 7) { ... }.  Putting it all on one line takes less mental effort to follow the code.

Threading

Both the producer and consumer threads are infinite loops. Neither thread will end normally, and therefore trying to .join() them is superfluous.
Rather than ensuring that the producer runs first, you should just design the consumer to correctly handle the case when there are no more values to consume.  As @DanielR pointed out, when list.size() is zero, r.nextInt(list.size()) throws an IllegalArgumentException.
Your use of lock.notify() right now is superfluous, as there is no corresponding lock.wait() that expects a notification.  A good place to insert a lock.wait() would be where you suspected.  (Without the lock.wait(), the consumer would keep trying frantically to fetch a value, which could work, but is inefficient.  Inserting lock.wait() makes the consumer yield the processor to the producer until the producer notifies the consumer that the values have been replenished.)

public void consumer() throws InterruptedException {
    synchronized (lock) {
        while (list.isEmpty()) {
            // Nothing to consume for now; wait for more.
            lock.wait();
        }
        ...
    }
}

I think that the vocabulary is slightly off.  To me, a "producer" and a "consumer" are both threads.  I would say,

Thread producer = new Thread(new Runnable() {
    @Override
    public void run() {
        low.produce();
    }
});

Therefore, I would rename your methods as verbs produce() and consume().
In Java, every Object can act as a lock.  There is no need to make a separate lock object; you could just synchronize on list or this instead.  (It doesn't matter which object you choose as the lock, as long as the producer and consumer both agree on which object it is.)  As a special case,

public void method() {
    synchronized (this) {
        ...
    }
}

can be written as

public synchronized void method() {
    ...
}

which is slightly more readable and also compiles to slightly tighter bytecode.
I would take move the while-loops from produce() and consume() into the run() method.  That would have the triple benefit of

keeping produce() and consume() simpler
making it more obvious that each Runnable is an infinite loop
moving the synchronized keyword into the method signatures, making it easier to see that produce() and consume() run mutually exclusively, and generating slightly tighter bytecode.

Proposed Solution

import java.util.ArrayList;
import java.util.Random;

public class LowLevelProducerConsumer {
    ArrayList<Integer> list = new ArrayList<Integer>();
    private final int SIZE = 10;
    Random r = new Random(System.currentTimeMillis());

    public synchronized void produce() {
        try {
            while (list.size() != SIZE) {
                list.add(r.nextInt(100));
            }
        } finally {
            // Notifies consumer that entries have been generated
            this.notify();
        }
    }

    public synchronized void consume() throws InterruptedException {
        while (list.isEmpty()) {
            // Nothing to consume for now; wait for more.
            // System.err.println("Consumer waiting...");
            this.wait();
        }

        if (r.nextInt(10) == 7) { // just an arbitrary condition
            int loc = r.nextInt(list.size());
            int data = list.remove(loc);
            System.out.format(
                    "%d removed from index %d , size : %d\n",
                    data, loc, list.size());
            Thread.sleep(300);
        }
    }

    public static void main(String[] args) {
        final LowLevelProducerConsumer low = new LowLevelProducerConsumer();
        Thread producer = new Thread(new Runnable() {
            @Override
            public void run() {
                while (true) {
                    low.produce();
                }
            }
        });
        Thread consumer = new Thread(new Runnable() {
            @Override
            public void run() {
                while (true) {
                    try {
                        low.consume();
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
            }
        });

        producer.start();
        consumer.start();
    }
}


Checking if a date comes after another in Java I need to test if a date is posterior to another in java, I just need to be sure that my approach is a good one

public class Date {
    private int day;
    private int month;
    private int year;

    public Date(int day, int month, int year) {
        this.day = day;
        this.month = month;
        this.year = year;
    }

    public int getDay() {
        return day;
    }

    public int getMonth() {
        return month;
    }

    public int getYear() {
        return year;
    }

    public boolean posterieurTo(Date dateToCompare){
        if (year > dateToCompare.getYear())
            return true;
        else if (month > dateToCompare.getMonth())
            return true;
        else if (day > dateToCompare.getDay())
            return true;
        else
            return false;
    }

}

The obvious

As mentioned in @mtj's comment, we have three different JDK APIs for doing this, so you really shouldn't have to resort to reinventing-the-wheel.

java.util.Date.after(Date)
java.util.Calendar.after(Object)
java.time.LocalDate.isAfter(ChronoLocalDate)

Naming

Ok, maybe this is some simple assignment that you need to do... how about the naming? You should try to avoid using the same class name from the JDK, as it will get messy understanding which class is actually being used. For example, the wrong usage of your class might lead to confusing toString() representation, or syntax errors when trying to do a Date.getTime().

Also, since you are already using English mostly, you should translate that French-sounding method name to the English equivalent too, e.g. isAfter(). Whether you should have your entire codebase in non-English is another debating topic altogether. :)

Validation

You should be checking whether day and month are valid or not... you wouldn't want your bespoke class to start representing the 99th day of the 555th month of the 18288th year... assuming we're still talking about the Greogorian calendar.

null check

Next, assuming you have renamed this class, you should consider handling null properly: do you want to throw a NullPointerException as it is currently, or return a default value?

Boolean expressions

Finally, you can chain the boolean expressions together using && and || so that the method returns true the moment any of the expression is true too:

return dateToCompare != null && // assuming returning false for null values
        year > dateToCompare.getYear() || (year == dateToCompare.getYear() &&
            (month > dateToCompare.getMonth() || (month == dateToCompare.getMonth() &&
                day > dateToCompare.getDay())));

Admittedly, this is getting hard to comprehend.

(thanks to @JS1's answer for the update)


Custom text binding handler I created this custom binding handler:

ko.bindingHandlers.line = {
    update: function (element, valueAccessor, allBindings) {
        // First get the latest data that we're bound to
        var value = valueAccessor();

        // Next, whether or not the supplied model property is observable, get its current value
        var valueUnwrapped = ko.unwrap(value);

        // Now manipulate the DOM element
        if (valueUnwrapped) {
            $(element).after('<br />');   // Make the element invisible
        }
        return ko.bindingHandlers.text.update(element, function () {
            return valueUnwrapped;
        })
    }
};

I'm wondering about the part where I do ko.bindingHandlers.text.update.  Is there a better way to solve this?

Is it the only way to update the DOM with ko.bindingHandler.text.update, or is there any more straightforward way to do it?

You can call it more directly using this:

return ko.bindingHandlers.text.update(element, valueAccessor);

The valueAccessor is already a function that returns the value, and update already uses unwrap internally, so there is no need to use the inline function.

Or more safely, preserving the other arguments to the handler, you can call it like this:

return ko.bindingHandlers.text.update.apply(this, arguments);

However, wrapping text binding handler is better done with a preprocessor, to ensure that other parts of the binding lifecycle are handled normally.

ko.bindingHandlers.line = {
    preprocess: function (value, name, addBinding) {
        addBinding('text', value);
        return value;
    },
    update: function (element, valueAccessor, allBindings) {
        if (ko.unwrap(valueAccessor())) {
            $(element).after('<br />');   // Make the element invisible
        }
    }
};

Here is a fiddle.


Practicing OOP with Student and Person I've just learned the OOP concept of abstraction, and I was wondering if I was implementing it correctly in my code and if you guys can point out any bad practices.

test1.java

package OOPConcepts;

import java.util.Scanner;
public class test1 {
    static public Scanner input = new Scanner(System.in);
    public static void main(String[] args) {
        String fName, lName;
        int age;
        Student newPerson = new Student();
        input.close();
    }
}

Person.java

package OOPConcepts;

public class Person
{
    static private int pNumber = 0;
    private int pID;
    public int age;
    private String type;
    public String fName, lName;

    Person()
    {
        pNumber++;
        pID = pNumber;

        System.out.println("Enter first name:");
        String fName = test1.input.next();
        System.out.println("Enter last name:");
        String lName = test1.input.next();
        System.out.println("Enter age:");
        int age = test1.input.nextInt();
        type = "person";
        describe(fName, lName, age);

        System.out.println("Made a " + type);
    }
    public void setType(String type){
        this.type = type;
    }
    public String getType()
    {
        return this.type;
    }
    public void describe(String fName, String lName, int age)
    {
        this.fName = fName;
        this.lName = lName;
        this.age = age;
        System.out.println("Name: " + this.fName + " " + this.lName + "|Age: " + this.age);
    }
}

Student.java

package OOPConcepts;

public class Student extends Person{
    static private int sNumber = 0;
    private int studentID;
    public String major;

    Student()
    {
        sNumber++;
        studentID = sNumber;
        setType("student");

        System.out.println("Who is also a " + getType());
        System.out.println("Enter major: ");
        String major = test1.input.next();
        this.major = major;
    }
    public void setStudentID(int idNumber)
    {
        this.studentID = idNumber;
    }
    public int getStudentID()
    {
        return this.studentID;
    }
}

What is OOP?

There is more to OOP than just class inheritance and your example shows painfully clearly why class hierarchy is almost always completely unable to represent the real world. If a student is a specialized type of a person, then a person can not be a student and a teacher at the same time (e.g. an assistant to a teaching professor). Also, that person can no longer be a part time employee at the cafeteria either, because the person's status has been locked into being a student and student only. Clearly, class inheritance is the wrong tool here. The relation should be composition:

public class Student {
    private final StudentId studentId;
    private final Person person;

    public Student(StudentId studentId, Person person) {
        this.studentId = studentId;
        this.person = person;
    }

    ...
}


Simple word translator taking advantage of Wikipedia This is my first Clojure script (simple word translator based on Wikipedia), and I guess there are things which could be simplified / done more idiomatic way. Specifically, I wonder if get-translations couldn't be done better (it produces map lang code => term form regexp sequence).

(ns wiki-translate
   (:require [clojure.contrib.http.agent :as h])
   (:require [clojure.contrib.string :as s])
   (:import (java.net URLDecoder URLEncoder))
)

(defn url-decode
  ([text] (URLDecoder/decode text "UTF-8"))
)

(defn url-encode
  ([text] (URLEncoder/encode text "UTF-8"))
)

(defn get-url
  ([lg term] (str "http://" lg ".wikipedia.org/wiki/" (url-encode term)))
)

(defn fetch-url
  ([url] (h/string (h/http-agent url)))
)

(defn get-translations
  ([cnt]  (apply sorted-map (flatten (for  [s (re-seq #"interwiki-([^\"]+).*wiki\/([^\"]+)\".*/a>" cnt)] [(s 1) (s 2)]))))
)

(defn translate
   ([term src-lg tgt-lg] (
    let [translations (get-translations (fetch-url (get-url src-lg term)))]
      (if (contains?  translations tgt-lg) (s/replace-str  "_" " " (url-decode (get translations tgt-lg))) "<NOT FOUND>")))
)

(defn prompt-read
  ([] (prompt-read ""))
  ([prompt] (print (format "%s: " prompt)) (flush ) (read-line))
)

(defn prompt-translate
  ([] (let [src-lg (prompt-read "Source language (en, fr, de ...)") tgt-lg (prompt-read "Target language  (en, fr, de ...)") term (prompt-read "Term to translate")]
  (println (str "\"" term "\" translated from " src-lg " to " tgt-lg " : " (translate term src-lg tgt-lg)))))
)

(while true (prompt-translate))

Don't use the multiple-arity syntax for defn unless you actually need to:

(defn url-decode
  [text]
  (URLDecoder/decode text "UTF-8"))

Your code could use some line breaks here and there. This is how clojure is usually indented:

(defn translate
  [term src-lg tgt-lg]
    (let [translations (get-translations (fetch-url (get-url src-lg term)))]
      (if (contains?  translations tgt-lg)
        (s/replace-str  "_" " " (url-decode (get translations tgt-lg)))
      "<NOT FOUND>")))

and:

(defn prompt-translate
  []
  (let [src-lg (prompt-read "Source language (en, fr, de ...)")
        tgt-lg (prompt-read "Target language  (en, fr, de ...)")
        term (prompt-read "Term to translate")]
    (println (str "\"" term "\" translated from " src-lg " to " tgt-lg " : " (translate term src-lg tgt-lg)))))

Instead picking out the regexp groups by index, you should use destructuring:

(defn get-translations
  [cnt]
  (apply sorted-map
    (flatten
      (for  [[_ a b] (re-seq #"interwiki-([^\"]+).*wiki\/([^\"]+)\".*/a>" cnt)]
        [a b]))))

... or better yet, I think, would be to do something like this:

(defn get-translations
  [cnt]
  (apply sorted-map
    (mapcat
      rest
      (re-seq #"interwiki-([^\"]+).*wiki\/([^\"]+)\".*/a>" cnt))))


How to make this Regex more flexible? I am creating a simple template engine that uses Regex to find special expressions, which are parsed and processed. They are enclosed in Ruby-style opening tags and have the format:

    <% label %>  OR:
    <% function(arg1, arg2, arg3) %>

However the problem is that I cannot have multiple entries on the same line, so I can't have this:

<% name %> - <% age %> - <% gender %>

Instead I have to write it like this:

<% name %> -
       <% age %> -
       <% gender %>

How can I fix this? Below I included the code for the function that handles this. Basically what it does is read from a file, split everything line by line, then iterate through each line and do a regex match and return the results.

public function content()
{
    if (!file_exists($this->path))
        return;

    /* 0 = before, 1 = entry #1, 2 = entry #2, 3 = after */
    $content = file_get_contents($this->path);
    $pattern = '/^(.*)\<\%\s*(.+)\s*\%\>(.*)$/';
    $lines = explode("\n", $content);

    foreach ($lines as $line)
    {
        // Pattern match the line
        preg_match($pattern, $line, $matches);

        $result = '';

        // If there are no matches, then don't parse it and add the raw string to output
        if (count($matches) == 0)
        {
            continue;
        }

        // Parse the matches and run the appropriate action
        $before = $matches[1];
        $after = $matches[3];
        $entry = $matches[2];
        $return = '';

        // Process the entry
        $return = $this->processEntry($entry);

        // Add the processed line to the final result if not empty
        $result = $before . $return . $after;
        if (!empty($result)) $final .= $result . "\n";
    }
    return $final;
}

What to you thing about:

<?=label ?>  OR:
<?=function(arg1, arg2, arg3) ?>

PHP is already a template engine

Just write a wrapper around a include call and your template system is ready to use and more flexible than anything you could write.

That said, why to you match single lines? Please have a look at preg_replace_callback.

/\<\%\s*(.+?)\s*\%\>/

If you want to put the whitespace trimming in the callback and as you don't need to escape <>%, you could even use

/<%(.+?)%>/


Accessing Tor using Python 2.7.x Below are two functions that work no problem in my current script. They are written to be ran in Python 2.7.x

def tor_browser_initialise():
        """ This function checks whether the Tor Browser is running. If it isn't,
        it will open the Tor Browser.
        """
        processlist = []
        for p in psutil.process_iter():
            try:
                process = psutil.Process(p.pid)
                pname = process.name()
                processlist.append(pname)
            except:
                continue
        if "tor.exe" not in processlist:
            process = subprocess.Popen(r"C:\Program Files (x86)\Tor Browser\Browser\firefox.exe", stdout=subprocess.PIPE)
            time.sleep(30)

def connect_tor(url):
    """ This function accepts a URl as an argument. It accesses the URL via TOR before
    returning the HTML source code to the function that called it. This function also
    uses random browser information.
    """
    LOCALHOST = "127.0.0.1"
    PORT = 9150
    useragent_list = ['Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0',
                   'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/29.0',
                   'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36',
                   'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36',
                   'Mozilla/5.0 (compatible; MSIE 10.6; Windows NT 6.1; Trident/5.0; InfoPath.2; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727) 3gpp-gba UNTRUSTED/1.0',
                   'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)',
                   'Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14',
                   'Mozilla/5.0 (Linux; U; Android 4.0.3; ko-kr; LG-L160L Build/IML74K) AppleWebkit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30']
    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, LOCALHOST, PORT)
    socket.socket = socks.socksocket
    request = urllib2.Request(url)
    request.add_header('User-Agent', random.choice(useragent_list))
    response = urllib2.urlopen(request)
    return response

I'd like to know if there is a more concise and Pythonic way of writing the two functions. I haven't listed the dependant libraries / modules at the beginning of the code, but it does work correctly.

Avoid bare except

Writing except: without specifying a precise exception is asking for trouble, as anything will be caught, silencing all possible  bugs, instead use: except MyExpectedKindOfException.

Reconsider the very long sleeping

The function tor_browser_initialise ends with time.sleep(30).

That is a lot of time to sleep. Are you 100% sure that any call to that function will want to sleep so much?

Much worse, the sleep is not documented, so the caller will see his program hang on for 30 seconds for no apparent reason!

Just remove the call to time.sleep and let the user decide if and how much he wants to sleep after calling the function.


Similar event handlers for buttons I have 3 simple similar event handler functions that I would like to refactor. Any suggestions?

private void btnBuildingList_Click(object sender, EventArgs e)
    {
        selectedExportType = (int)ExportType.Building;
        path = csvFilePath + String.Format("{0:yyyy-MM-dd}", datDate.DateTime) + "-BuildingList.csv";
        Export();
    }

    private void btnOwnerList_Click(object sender, EventArgs e)
    {
        selectedExportType = (int)ExportType.Persons;
        path = csvFilePath + String.Format("{0:yyyy-MM-dd}", datDate.DateTime) + "-OwnerList.csv";
        Export();
    }

    private void btnFacts_Click(object sender, EventArgs e)
    {
        selectedExportType = (int)ExportType.Facts;
        path = csvFilePath + String.Format("{0:yyyy-MM-dd}", datDate.DateTime) + "-FactsData.csv";
        Export();
    }

You can use an Extract Method refactoring to get rid of the duplicate code:

private static string GetExportFilePath(string csvFilePath, DateTime date, string fileSuffix)
{
    return string.Format("{0}{1:yyyy-MM-dd}-{2}.csv", csvFilePath, date, fileSuffix);
}

I would add those selectedExportType and path variables as parameters to the Export() method. I hope they are not used anywhere. It is usually pretty subjective whether to add something as method parameters or leave it as class member, but here I would definitely pass them as parameters. I assume this class is a “View” since you have button click event handlers, and I also tend to move such methods out from the views.

If we use these two ideas, your event handlers would be simplified to:

private void btnBuildingList_Click(object sender, EventArgs e)
{
    var path = GetExportFilePath(csvFilePath, datDate.DateTime, "BuildingList.csv");
    Export((int)ExportType.Building, path);
}

private void btnOwnerList_Click(object sender, EventArgs e)
{
    var path = GetExportFilePath(csvFilePath, datDate.DateTime, "OwnerList.csv");
    Export((int)ExportType.Persons, path);
}

private void btnFacts_Click(object sender, EventArgs e)
{
    var path = GetExportFilePath(csvFilePath, datDate.DateTime, "FactsData.csv");
    Export((int)ExportType.Facts, path);
}


Recursive function for bubble sort I often see nested loops as a solution to the bubble sort, but my solution involves a recursive function. Is the following solution less efficient than the traditional method?
public static int[] BubbleSort(int[] arrayOfValues)
{
    var swapOccurred = false;

    for (var i = 0; i < arrayOfValues.Length; i++)
    {
        if (i == arrayOfValues.Length - 1)
            continue;

        if (arrayOfValues[i] > arrayOfValues[i + 1])
        {
            //swap values
            var current = arrayOfValues[i];
            var next = arrayOfValues[i + 1];

            arrayOfValues[i] = next;
            arrayOfValues[i + 1] = current;

            swapOccurred = true;
        }
    }

    if (swapOccurred)
    {
        // keep going until no further swaps are required:
        BubbleSort(arrayOfValues);
    }

    return arrayOfValues;
}

Less efficient? It consumes more stack space, which shouldn’t be a problem for any dataset where a bubble sort would be appropriate.
With the latest version of the c# compiler, tuple deconstruction will allow you to swap the values with a single line of code (although it will probably be slower).
On a side note, I’m not clear on why you have a separate continue instead of having a lower conditional check. It’s the same thing.
Not sure if you know this, but you are examining already sorted elements with your recursive call. An “efficient” bubble sort moves the end position back one cell after each cycle.
Update: I did some rough testing your implementation, and get a stack overflow a bit after 8k items. Sorting 8k random numbers without reducing the end point took ~.75 seconds, sorting with reducing the end point took ~.70 seconds. Much better than I expected.  Using Moch Yusup modification to use a more traditional structure the time for 8k items was approximately the same, but it took 116.8 seconds for 80k items.
So, performance wise, within the usable range, recursion was functionally identical, but due to stack overflow the usable range is limited.  OTOH, the usable range and "not enough time to go get a cup of coffee" range are fairly close.
Just for comparison, I tried using the standard List.Sort method, which took .0003681 seconds for 8k items, and .006 seconds for 80k items...


Performance of unpack() I wrote a PHP class in Zephir to increase performance, but it is still very slow and memory consuming. To compare, I wrote several variants of the same code, but unpack() is always nearly as slow at storing the data in a csv file.

Is there any way to optimize this code? About 30% of the whole program time is spent in unpack.

class Hdb implements \Iterator, \Countable {
    /**
     * File handle.
     * @var resource
     */
    protected fh;

    protected line_count = 0;
    protected position = 0;

    protected row = [];

    public function __construct(string file) -> void {
        let this->line_count = floor((filesize(file)) / 12);
        let this->fh = fopen(file, "r");
    }

    public function readLine() {
        var timestamp;
        var value;
        var status;

        let timestamp = unpack("L", fread(this->fh, 4));
        let value     = unpack("f", fread(this->fh, 4));
        let status    = unpack("I", fread(this->fh, 4));

        let this->position++;

        return [
            "timestamp": timestamp[1],
            "value":     value[1],
            "status":    status[1]
        ];
    }

    // ... The standard Iterator stuff.

Gist

I think your performance issue is caused by going to file multiple times. The more you read in at once the less time you'll spend asking the operating system to provide you with a few more bytes for unpack to operate on.

Similar to the above answer:

$data = fread(fh,12);
$a = unpack("L",substr($data,0,4));
$b = unpack("f",substr($data,4,4));
$c = unpack('I',substr($data,8,4));

If you want to reorganize your code you could load the entire file and then work on the data within it.

$packed = file_get_contents("somefile.dat");
$len  = strlen($packed);

$offset = 0;
while ( $offset < $len )
{
  $a[] = unpack("L",$packed,$offset,4);
  $b[] = unpack("f",$packed,$offset+4,4);
  $c[] = unpack("I",$packed,$offset+8,4);
  $offset += 12;
}

Of course, you may have to reorganize various sections of code to support working on larger chunks of the input data at a time.

Note: It appears that unpacked is being unfairly blamed for being slow due to the filesystem access embedded in the function call. You might also want to do a few tests to clarify what is taking time in the small chunk method and see how much it speeds up based on increasing the read size used (if you have the time and inclination).

EDIT: Adding the following after comments.

You are right. I did some tests and while reading more data at once does save time it isn't as significant as reducing calls to unpack.

Running unpack on binary file (12 byte reads, 3 unpacks per read)...
... fread time: 3.5979931354523
... unpack time: 12.179003238678
... total measured time: 15.77699637413
... records: 265533

Running unpack on binary file (preload file, 3 unpacks per 12 bytes)...
... fread time: 0.016999959945679
... unpack time: 12.446995973587
... total measured time: 12.463995933533
... records: 265533

Running unpack on binary file (preload file, 12 bytes per unpack)...
... fread time: 0.01200008392334
... unpack time: 4.1770012378693
... total measured time: 4.1890013217926
... records: 265533

Running unpack on binary file (preload, 24 bytes per unpack)...
... fread time: 0.010999917984009
... unpack time: 2.189001083374
... total measured time: 2.200001001358
... records: 265534 (* padding added at end of input)

Run using PHP with microtime(true) capture before and after relevant actions.

EDIT: With a bit more fooling around I can now give you an "answer" regarding how to increase performance.

Split your data internally in order to get as much work as possible out of a single call to unpack.

if ( 1 )
{
  echo "
\n";
  echo "Running unpack on binary file (preload, split into 3 streams, unpack each stream)...
\n";
  ob_flush();
  flush();

  $records = 0;
  $fread_time = 0.0;
  $unpack_time = 0.0;
  $measured = 0.0;

  $start = microtime(true);
  $data  = file_get_contents($bin_file);
  $size  = filesize($bin_file);
  $stop  = microtime(true);
  $fread_time = $stop - $start;

  $Ldata = '';
  $fdata = '';
  $Idata = '';
  $offset = 0;
  $start = microtime(true);
  while ($offset < $size)
  {
    $Ldata .= substr($data,$offset,4);
    $offset += 4;
    $fdata .= substr($data,$offset,4);
    $offset += 4;
    $Idata .= substr($data,$offset,4);
    $offset += 4;

    $records++;
  }
  $stop = microtime(true);
  $split_time = $stop - $start;

  $start = microtime(true);
  $a = unpack('L*',$Ldata);
  $b = unpack('f*',$fdata);
  $c = unpack('I*',$Idata);
  $stop = microtime(true);
  $unpack_time += ($stop - $start);
  $measured_time = $fread_time + $split_time + $unpack_time;

  echo "... fread time: $fread_time
\n";
  echo "... split_time: $split_time
\n";
  echo "... unpack time: $unpack_time
\n";
  echo "... total measured time: $measured_time
\n";
  echo "... records: $records
\n";
  ob_flush();
  flush();
}

This provides a much better time.

Running unpack on binary file (preload, split into 3 streams, unpack each stream)...
... fread time: 0.013000011444092
... split_time: 0.29400014877319
... unpack time: 0.54299998283386
... total measured time: 0.85000014305115
... records: 265533

Hopefully this is a bit more useful.


Send / receive wrapper for System.Net.Sockets The following NetworkEndpoint class serves as a simple send / receive interface for either side of a network connection (client or server) when provided with a socket to operate through.

I have two main concerns about the code in it's current state:

What can I do to improve to exception handling to better deal with the socket becoming disconnected (both unexpectedly and intentionally)?
As far as I can tell I have protected against multiple receive loops begin started. (if (Receiving) return; in void InitializeReceiveLoop()) but are there any other major thread safety issues I have overlooked??

My knowledge of C# is primarily self taught so I apologize if my code does't reflect best practices or standard style guides.

Any advice on either of these issues or advice in general on improving my code would be greatly appreciated.

public class NetworkEndpoint {

    private Socket connection;
    public bool Connected { get => connection != null ? connection.Connected : false; }
    public bool Receiving { get; private set; }

    public event EventHandler<NetworkReceiveEventArgs> DataReceived;
    private void OnDataReceived(NetworkHeader header, byte[] data) =>
    DataReceived?.Invoke(this, new NetworkReceiveEventArgs(header, data));

    public event EventHandler Disconnected;
    private void OnDisconnected() =>
    Disconnected?.Invoke(this, new EventArgs());

    public NetworkEndpoint(Socket socket) {
        connection = socket ?? throw new ArgumentNullException("socket");
        Receiving = false;
    }

    public void Send(NetworkHeader header, byte[] data) {
        if (!Connected) throw new InvalidOperationException("NetworkEndpoint must be connected before sending.");
        try {
            connection.Send(ByteUtils.Combine(header.Serialize(), data));
            } catch (SocketException ex) {
            Disconnect();
        }

    }

    public void Disconnect() {
        connection?.Shutdown(SocketShutdown.Both);
        connection?.Close();
        connection = null;
        OnDisconnected();
    }

    public void InitializeReceiveLoop() {
        if (Receiving) return;

        Receiving = true;
        BeginReceive();
    }

    private byte[] headBuffer = new byte[NetworkHeader.ByteSize];
    private void BeginReceive() {
        connection.BeginReceive(headBuffer, 0, NetworkHeader.ByteSize, SocketFlags.None, EndReceive, null);
    }

    private void EndReceive(IAsyncResult result) {
        bool fullyReceived = false;
        byte[] dataBuffer = null;
        NetworkHeader header = null;

        try {
            if (connection.EndReceive(result) > 0) {
                header = NetworkHeader.Deserialize(headBuffer);
                dataBuffer = new byte[header.DataSize];

                int receivedLength = connection.Receive(dataBuffer, 0, dataBuffer.Length, SocketFlags.None);
                fullyReceived = (receivedLength == header.DataSize);
            }
        } catch (SocketException) { }

        if (fullyReceived) {
            OnDataReceived(header, dataBuffer);
            BeginReceive();
        } else if (connection.Connected) Disconnect();
    }
}

Review

Property Connected is not thread-safe. Use Interlocked.CompareExchange to get the connection atomically.
Events DataReceived and Disconnected are never cleared. This will cause a memory leak. Implement a Clear function and/or implement IDisposable.
Methods Send, Disconnect, EndReceive are not thread-safe. Consider using a mutex/lock.
Checking fullyReceived on receivedLength == header.DataSize is naive and can be broken. TCP messages can be sent in several packets. Use an internal buffer instead. Also, receivedLength = 0 is a clean way of determining no more data is available.
Catching SocketException blindly is a code-smell.


Kattis, Speed limit; read irregular input I am able to solve this challenge but when look at my code I know it is not good.
The challenge:

Bill and Ted are taking a road trip. But the odometer in their car is broken, so they don’t know how many miles they have driven. Fortunately, Bill has a working stopwatch, so they can record their speed and the total time they have driven. Unfortunately, their record keeping strategy is a little odd, so they need help computing the total distance driven. You are to write a program to do this computation.
For example, if their log shows
Speed in miles per hour | Total elapsed time in hours
20                     | 2
30                     | 6
10                     | 7
this means they drove 2 hours at 20 miles per hour, then 6−2=4 hours at 30 miles per hour, then 7−6=1 hour at 10 miles per hour. The distance driven is then 2⋅20+4⋅30+1⋅10=40+120+10=170 miles. Note that the total elapsed time is always since the beginning of the trip, not since the previous entry in their log.
Input:
The input consists of one or more data sets. Each set starts with a line containing an integer n, 1≤n≤10, followed by n pairs of values, one pair per line. The first value in a pair, s, is the speed in miles per hour and the second value, t, is the total elapsed time. Both s and t are integers, 1≤s≤90 and 1≤t≤12. The values for t are always in strictly increasing order. A value of −1 for nsignals the end of the input.
Output:
For each input set, print the distance driven, followed by a space, followed by the word “miles”.

Sample input:
3
20 2
30 6
10 7
2
60 1
30 5
4
15 1
25 2
30 3
10 5
-1

Sample output:
170 miles
180 miles
90 miles

I try to split the input to case-by-case group and solve it using calculate but end-up with deep nested list.
import Data.List.Split

main :: IO ()
main = do
  as <- map words . lines <$> getContents
  let bs = filter (not.null) $ splitWhen ((==1).length) as
  let cs = map (map (map (read :: String -> Int))) bs
  let ds = map calculate cs
  mapM_ putStrLn ds

calculate :: [[Int]] -> String
calculate xs = z
  where as = map head xs
        bs = map last xs
        cs = head bs : zipWith (flip (-)) bs (tail bs)
        ds = zipWith (*) as cs
        z = show (sum ds) ++ " miles"

The code is ugly especially on line 7: map (map (map (read :: String -> Int)))
There are lot of let and meaningless bindings (as, bs, ...) because I need to insert print xs to see what the list is look like after each step.
Reading code again without print output, I hardly get the picture what the list structure is. This can't be good.
How should I make it better?

Your approach to the problem seems very imperative and whole-problem focused. Thinking in Haskell generally benefits from considering more bottom-up strategies, using smaller, easily tested, reusable pieces.

Right off the bat, your representation of the core datatype (a log of speeds and timestamps) seems ill-fitted. A list of list of Ints doesn't capture the fact that you know you will only be receiving pairs of numbers at a time.

type Speed = Int
type Timestamp = Int
type LogLine = (Speed, Timestamp)
type Log = [LogLine]

-- Less vitally...
type Logs = [Log]

Type aliases help provide documentation to readers of your code (including you as you write it, or come back to it after any significant stretch of time), use them liberally when working with the basic datatypes. The above set of aliases captures the input as it will be given to you (post- the act of parsing). Lists of pairs of numbers in the string input become []s of (,)s of Ints in Haskell.

Next we have to determine what sort of format we want to transform the data into in order to calculate an answer from it. The problem would be simple if we could replace those timestamps with durations, so we'll figure out a way to do that.

type Duration = Int

durations :: [Timestamp] -> [Duration]

Note that speeds won't be relevant when converting timestamps to durations, but we will need all of the timestamps in a log in order.

Once we have speed and duration pairs, solving for the solution should be easy.

type Distance = Int

calculate :: [(Speed, Duration)] -> Distance
calculate = sum . map (uncurry (*))

And if we hop in GHCi and test that definition with some hand-crafted data (i.e., the worked example given in the problem statement)—

λ sum . map (uncurry (*)) $ [(20, 2), (30, 4), (10, 1)]
170

Looks good. Now we work backward, filling in the pieces we left undefined, and combining them together as appropriate.

durations :: [Timestamp] -> [Duration]
durations ts = zipWith (-) ts (0:ts)

solve :: Log -> Distance
solve ls =
  let (speeds, timestamps) = unzip ls
  in  calculate $ zip speeds (durations timestamps)

With a little more consideration (and certainly if we were willing to pull in the lens library) we could code golf the unzip/zip away and end up with a pithy one liner for solve, but often the most straightforward solution is more than adequate.

Now we need only deal with the parsing and pretty-printing aspects of the problem. Given that all of these code challenge tasks are similar, much of this is boilerplate you reuse from problem to problem, just make sure you're familiar with the contents of the Control.Monad module.

main :: IO ()
main = do
  n <- readLn
  if n == -1
  then return () -- End flag, don't recurse
  else do
    log <- replicateM n $ do
      [speed, timestamp] <- map read . words <$> getLine
      return (speed, timestamp)
    let distance = solve log
    putStrLn $ (show distance) ++ " miles"
    main -- recurse, look for the next trip


FTP Download Function This is an function that downloads a single file from an FTP server and saves it to disk
This is my first time doing anything with FTP, please let me know if i have missed anything.

public void DownloadFile(string fileloc, string saveLoc)
{
    try
    {
        // Get the object used to communicate with the server.
        FtpWebRequest request = (FtpWebRequest)WebRequest.Create(fileloc);
        request.Method = WebRequestMethods.Ftp.DownloadFile;

        request.Credentials = new NetworkCredential(Username, Password);

        using (var response = (FtpWebResponse)request.GetResponse())
        {
            using (var fileStream = File.Create(saveLoc))
            {
                response.GetResponseStream().CopyTo(fileStream);
            }
        }
    }
    catch (Exception EX)
    {
        throw EX;
    }
}

First, like ANeves said, the only thing your catch does is that it resets the exception's stack trace, which is not what you want to do. Just remove the try/catch completely, it serves no purpose here.

Second, I don't see any reason to use WebRequest here, using WebClient is much simpler.

Also, you might want to have better parameter names: fileloc doesn't say what exactly it is, and doesn't use the usual capitalization. And loc is unnecessarily abbreviated (you're not trying to write a Twitter post where every character counts, readability is more important) and too ambiguous.

public void DownloadFile(string url, string savePath)
{
    var client = new WebClient();
    client.Credentials = new NetworkCredential(Username, Password);
    client.DownloadFile(url, savePath);
}

This method could be rewritten as one expression, but I probably wouldn't do that:

public void DownloadFile(string url, string savePath)
{
    new WebClient { Credentials = new NetworkCredential(Username, Password) }
        .DownloadFile(url, savePath);
}


Populating an array from a database I have a query that populates an array from the database. In some cases, this query returns a great amount of data, (let's say for purpose of an example, 100.000 records). Each row of the database has at least 6 or 7 columns.

$results = [
  ['id' => 1, 'name' => 'name', 'status' => true, 'date' => '10-01-2012'],
  ['id' => 2, 'name' => 'name 2', 'status' => false 'date' => '10-01-2013'],
  ...
]

I need to perform a substitution of some of the data inside the $results array, based on another one that give me some information about how i would change the values in the rows.

$model = [
    'status' => ['function' => 'formatStatus', params => ['status']],
    'date' => ['function' => 'formatDate', params => ['date']]
]

Now that I have all the data, what do I do with it? I have the following routine:

foreach ($results as &$itemResult) {
        $oldValues = $itemResult;
        foreach ($itemResult as $attribute => &$value) {
            if (isset($model[$attribute]['function'])) {
                $function = $model[$attribute]['function'];
                $params = $model[$attribute]['params'];
                $paramsSize = count($params);
                for ($i = 0; $i < $paramsSize; $i++) {
                    $newParams[] = $oldValues[$params[$i]];
                }
                $itemResult[$attribute] = call_user_func_array([$this, $function], $newParams);
                $newParams = null;
            }
        }
    }

So, for each attribute for each row of my data array, I run check for the existence of a function and params information. When the attribute in question needs to be replaced, I call the function via call_user_func_array and replace the value with the function return value.

Also notice that I am replacing the current array, not creating another, by passing the reference &$itemResult inside the loop, so in the end, I have the same array from the beginning but with all columns that needed to be replaced with its new values.

The thing is, for little arrays, this method is quite good. But for big ones, it becomes a pain. I do believe that an array this size consumes so much RAM that makes the whole algorithm slow.

Could you provide me with some alternative to the problem? Should I use another data structure instead of the PHP array?

As noticed by mmm a solution might be (if applicable) to slice results. Anyway, here is my suggestion for the process to be as thrifty as possible.

// First create a specific model version appropriate to current results structure
$curModel = [];
$resultSample = reset($results);
foreach ($model as $attribute => $ope) {
  // keep only useful attributes
  if (array_key_exists($attribute, $resultSample) {
    // flip params list to have param as key
    $curModel[$attribute] = [
      'function' => $ope['function'],
      'params' => array_flip($params),
    ];
  }
}

// Then work:
// . without having to check if some attribute applies
// . with the ability to use array_intersect_key() to provide params
foreach ($results as &$itemResult) {
  foreach ($curModel as $attribute => $ope) {
    $function = $ope['function'];
    $itemResult[$attribute] =
      $function(array_intersect_key($itemResult, $ope['params']));
  }
}

It's based on the attempt to move as much work as possible, from the loop where it's repetitively executed, to a preparatory work executed once:

build a version of $model strictly suitable for the current results structure (so we don't have to check it further)
invert the nested iterations: rather than iterate on $model attributes and look if applicable, now we know that the $model contains only applicable attributes, so for each one we can directly work with the current record contents

At the same time, I chose to simplify the transformer function invocation: passing a unique array argument avoids using call_user_func_array(). Note that it needs the adaptation of the transformer functions, but it should be quite easy.

A few more little things:

I stopped using a temporary copy of $itemResult, which is useless
I used as much PHP specific functions as possible (instead of a sequence of statements) to reduce processor consumption: array_flip(), array_intersect_key()


Safely using android 'Context' inside threads Oftentimes I find myself using context inside threads, multiple times. The problem is that I do not want to hold a long-lived strong reference of it in my threads to avoid leaking it. I keep a weakReference to it in global space and when my activity dies, it will obviously not prevent any GC events or such.

I have made the following approach to using context in threads:

public interface UseContext<Result, Param extends Context> {
    Result work(Param context);
}

public static <T extends Context, Result> Optional<Result> useContext(final WeakReference<T> reference,
                                                                      final UseContext<Result, T> task) {

    final T context;
    final boolean contextLost = (context = reference.get()) == null;
    final boolean isActivity = context instanceof Activity;

    if (contextLost)
        return Optional.absent();
    final boolean activityIsFinishing = (isActivity && ((Activity) context).isFinishing());
    if (activityIsFinishing)
        return Optional.absent();

    return Optional.fromNullable(task.work(context));
}

Whenever I have to use context inside a thread, I do this (example) :

        MiscUtils.useContext(reference, activity -> {
            activity.runOnUiThread(() -> Toast.makeText(activity, message, Toast.LENGTH_SHORT).show());
            return null;
        });

Is this a correct approach? Also, I'm not entirely convinced that holding a strong reference to Context in global space and using it in runnable is a bad idea. I understand that it was something to do with anonymous inner classes causing memory leaks.

I think it's a bit of overhead.

Depending on what you need to do with the Context, you can the Application Context instead of the Activity itself. This context is available while the process runs, so there no leakage is possible.

In my experience, Activities being leaked have been caused by singletons initialized by passing the Activity instance (as a Context). Be careful also with Views, as they hold a reference to the Context.

Using the StrictMode with the setClassInstanceLimit and a bit of logging has helped me catch these leaks quite fast.


Polynomial curve-fitting over a large 3D data set I have a list of 4 images, called listfile.list, which looks like this:

image1
image2
image3
image4

Each image has 10 frames containing a 2000 x 2000 array of pixels, so the size of each image is [10,2000,2000]. The pixel value for each frame increases from 0 to 10, so for example for one pixel in image1:

image1[:,150,150] = [435.8, 927.3, 1410. , 1895.1, 2374.6, 2847.1,
                     3340.5, 3804.8, 4291.6, 4756.1]

The other pixels show similar values and a roughly linear increase across the frames. I need to fit several functions to pixel values across the frames for each pixel in each image and then average over the images. The data sets are 3D and very large, so I don't know how to paste them here. My code does exactly what I want it to, the issue is that it can take days (> 3) to run my full script.

One function is frame_fit to return rates and intercepts. There are several other functions. My code is structured as follows:

import itertools
import numpy as np
from scipy.optimize import curve_fit

def frame_fit(xdata, ydata, poly_order):
    '''Function to fit the frames and determine rate.'''

    # Define polynomial function. Here, 'b' will be ideal rate and
    # 'c', 'd', 'e', etc. describe magnitudes of deviation from
    # ideal rate.
    if poly_order == 5:
        def func(t, a, b, c, d, e, f):
            return a+ b*t+ c*(b*t)**2+ d*(b*t)**3+ e*(b*t)**4+ f*(b*t)**5

    elif poly_order == 4:
        def func(t, a, b, c, d, e):
            return a + b*t + c*(b*t)**2 + d*(b*t)**3 + e*(b*t)**4

    # Initial values for curve-fitting.
    initvals = np.array([100, 4.e+01, 7.e-03, -6.e-06, 3.e-08, -8.e-11])

    # Provide uncertainty estimate
    unc = np.sqrt(64 + ydata)

    beta, pcov = curve_fit(func, xdata, ydata,
                           sigma=unc, absolute_sigma=False,
                           p0=initvals[:poly_order+1],
                           maxfev=20000)

    # beta[1] is rate, beta[0] is intercept

    return beta[1], beta[0]

all_rates = np.zeros((number_of_exposures, 2000, 2000),dtype=np.float64)
all_intercepts = np.zeros((number_of_exposures, 2000, 2000),dtype=np.float64)
all_results = np.zeros((2, 2000, 2000),dtype=np.float64)
pix_x_min = 0
pix_y_min = 0
pix_x_max = 2000    # max number of pixels in one direction
pix_y_max = 2000
xdata = np.arange(0,number_of_frames)    # so xdata [0,1,2,3,4,5,6,7,8,9]

# Here is where I need major speed improvements

for exposure in list_of_exposures:

    for i,j in itertools.product(np.arange(pix_x_min,pix_x_max),
                                     np.arange(pix_y_min,pix_y_max)):

        ydata = exposure[:,i,j]

        rate, intercept = frame_fit(xdata,ydata,5)

        # Plus other similar functions ...
        # results2, residuals2 = function2(rate, intercept)
        # results3, residuals3 = function3(results2, specifications)

        all_rates[exposure,i,j] = rate
        all_intercepts[exposure,i,j] = intercept

avg_rates = np.average(all_rates, axis=0)
avg_intercepts np.average(all_intercepts, axis=0)

all_results[0,:,:] = avg_rates
all_results[1,:,:] = avg_intercepts

# all_results is saved to a file after everything is finished.

There are usually at least 4 images with 2000x2000 arrays of pixels in my input list. Besides the frame_fit function, there are several others that I have to run for each pixel. I am a relatively new Python programmer so I often don't know about all available tools or best practices to improve speed. I know a nested for-loop as I've written it is going to be slow. I have considered cython or multiprocessing, but those are complicated to tackle for a new programmer. Is there a better way to perform a function on each pixel and then average over the results? I am hoping to stick with standard python 3.5 modules (rather than installing extra packages).

In terms of possible speed optimisations there's not much to work with, but one thing does stand out:

        def func(t, a, b, c, d, e, f):
            return a+ b*t+ c*(b*t)**2+ d*(b*t)**3+ e*(b*t)**4+ f*(b*t)**5

(and similarly for the four-element case). It's fairly standard knowledge that polynomials evaluation can be optimised for speed using Horner's method. I'll also pull out that multiplication by b:

        def func(t, a, b, c, d, e, f):
            bt = b*t
            return a + bt*(1 + bt*(c + bt*(d + bt*(e + bt*f))))


Are these if-statements too fancy? I just rewrote this:

if (budgetRemaining != 0 || totalOpenInvoices != 0)
{
}

Like this:

if (new[] { budgetRemaining, totalOpenInvoices }.Any(c => c != 0))
{
}

If I had seen that before I ramped up on Linq, it would have confused me.  Now that I've been learning functional programming and using Linq, it seems natural, but is it sacrificing simplicity?

Seems to be swatting a fly with a Buick to me. The first form seems pretty concise and the variable names are quite descriptive. The second form creates a new object (the array) which will eventually have to be GC'd and introduces a new lambda variable, c which doesn't seem descriptive any more.


Parsing key:value declarations I recently ran into some issues with a plugin and outlined the issue in this post and wanted to know if the modifications I've made below cover the scenarios that the with scope blocks would have covered.

I've modified some code to remove the with operator and I'm wondering if I've replicated everything properly in doing so.

var test = new Function('$f','$c','with($f){with($c){return{'+ declarations +'}}}'));

Where $f and $c are passed objects (From what I could tell, $f shouldn't ever have a property of $c). The declarations variable is a string that has a colon in it (EX: "value:color") and available within the scope.

Here is my modified code:

var test = function($f, $c, declarations) {
        var result = {};
        var value = "";
        var split = declarations.split(":");
        if (split.length < 2) {
          throw new Error("Declaration is in an invalid format");
        }
        if ($f[$c] !== undefined && $f[$c][split[1]]) {
          value = $f[$c][split[1]];
        }
        else if ($c[split[1]]) {
          value =  $c[split[1]];
        }
        else if ($f[split[1]]) {
          value =  $f[split[1]];
        }
        else {
          value =  "" + split[1];
        }
        var key = split[0];
        result[key] = value;
        return result;
};

Everything appears to work as it did previously, but this modification now handles the use case where the declarations variable could have a dash in it (EX: "value:background-color").  Additionally the declarations variable is passed into the function, to ensure it's defined.

Consider tightening up the long if statement using || which should be equivalent to your current logic.  I can't test this for you, but see if this works:

var test = function($f, $c, declarations) {
  var split = declarations.split(":"), key, value, cIsFProp;

  if (split.length < 2)
    throw new Error("Declaration is in an invalid format");

  cIsFProp = $f[$c] !== undefined && $f[$c][split[1]];

  value = cIsFProp ? $f[$c][split[1]] : null;
  value = value || $c[split[1]]) || $f[split[1]]) || "" + split[1];
  key = split[0];
  return {[key]: value};
};


Project Euler 67 | Maximum sum path II reading from file I tried to solve the Problem Euler 67 with golang, because I started to study Go recently.
This is my solution:
package main

import (
    "flag"
    "fmt"
    "io/ioutil"
    "os"
    "strconv"
    "strings"
    "time"
)

func calculateMaxPath(trianglePointer *[][]int, lines int) int {
    trisums := make([][]int, lines)
    triangle := *trianglePointer

    for row := lines - 1; row > -1; row-- {
        trisums[row] = make([]int, row+1)
        for col := 0; col < row+1; col++ {
            if row == lines-1 {
                trisums[row][col] = triangle[row][col]
            } else {
                if trisums[row+1][col] > trisums[row+1][col+1] {
                    trisums[row][col] = triangle[row][col] + trisums[row+1][col]
                } else {
                    trisums[row][col] = triangle[row][col] + trisums[row+1][col+1]
                }
            }
        }
    }

    return trisums[0][0]
}

func openFile() (*[][]int, int) {
    flag.Parse()

    ifile := flag.Arg(0)

    if len(ifile) == 0 {
        println("No File Specified")
        os.Exit(1)
    }

    content, err := ioutil.ReadFile(ifile)

    if err != nil {
        println("Error reading", ifile)
        os.Exit(1)
    }

    strRows := strings.Split(string(content), "\n")
    content = nil

    lines := 0
    for i := len(strRows) - 1; i > 0; i-- {
        if lines == 0 && strRows[i] != "" {
            lines = i + 1
            break
        }
    }

    var strs []string

    pyramid := make([][]int, lines)
    for row := 0; row < lines; row++ {
        strs = strings.Fields(strRows[row])
        pyramid[row] = make([]int, len(strs))

        for col := 0; col < len(strs); col++ {
            pyramid[row][col], _ = strconv.Atoi(strs[col])
        }
    }

    return &pyramid, lines
}

func main() {
    start := time.Now()

    pyramid, lines := openFile()

    greatestSum := calculateMaxPath(pyramid, lines)

    fmt.Printf("\nTotal sum: %v, Elapsed time: %v\n", greatestSum, time.Since(start))
}

What could be a better approach? Coud I use go routines to increase performance?

I tried to solve the Problem Euler 67 with Go, because I started to
study Go recently.
What could be a better approach? Could I use goroutines to increase
performance?

Code should be correct, maintainable, robust, reasonably efficient, and, most importantly, readable.
I'm going to skip a full code review and narrowly focus my code review on the use of performance benchmarks. Go has several tools to measure performance, including the Go standard library testing package Benchmark type.

The elapsed time for the program is somewhat interesting but not very useful. Some results from running my program code and your program code several times to prime the Linux file buffer cache:
peterGo:
$ go run euler67.go
7273
175.641µs

Mdsp:
$ go run mdsp.go
Total sum: 7273, Elapsed time: 277.556µs

Here's how I benchmarked your Project Euler Problem 67 maximum sum path algorithm using p067_triangle.txt as input.
BenchmarkEuler67 measures overall (load and sum) performance while ignoring the effects of hardware (for example, HDD I/O), operating system (for example, cached disk data), and other extraneous effects. The triangle has 100 rows with a total of 5050 (= 100 * (100 + 1) / 2) elements.
$ go test euler67_test.go euler67.go -bench=. -benchmem
BenchmarkEuler67-4   13299   89766 ns/op   68440 B/op   210 allocs/op

These are benchmarks for BenchmarkEuler67 components.
$ go test euler67_test.go euler67.go -bench=. -benchmem
BenchmarkLoad-4      16959   70742 ns/op   68440 B/op   210 allocs/op
BenchmarkRow-4     1000000    1137 ns/op   896 B/op       1 allocs/op
BenchmarkSum-4       60830   19595 ns/op   0 B/op         0 allocs/op

BenchmarkLoad measures loading the in-memory text triangle values into a triangle of int slices.
BenchmarkRow measures parsing the text for the maximum (100 integer) triangle row into to a slice of Go ints. parseInts, a specialization, provides an improvement over strings.Fields for the loadTriangle function.
strings.Fields:
BenchmarkLoad-4      10000  114958 ns/op  153304 B/op   310 allocs/op

parseInts:
BenchmarkLoad-4      16959   70742 ns/op   68440 B/op   210 allocs/op

BenchmarkSum measures the fundamental maximum sum path algorithm performance: 19.6 microseconds. The maxPathSum function reduces memory bytes and allocations to zero by reusing the input for intermediate results.
Clearly, the major expense is loading the inefficient input into an efficient data structure for the maximum sum path algorithm.
To see how the code scales, here are benchmarks for a triangle with 1000 rows with a total of 500500 (= 1000 * (1000 + 1) / 2) elements.
BenchmarkEuler67-4     174   6810618 ns/op   5941675 B/op   2015 allocs/op
BenchmarkLoad-4        186   6399151 ns/op   5941685 B/op   2015 allocs/op
BenchmarkRow-4       93747     12505 ns/op      8192 B/op      1 allocs/op
BenchmarkSum-4        1916    622193 ns/op         0 B/op      0 allocs/op

6.8 milliseconds for 1000 rows is not unreasonable.
Improving performance is an iterative process and there is more that we can do, but is it worth it. The CPU, memory, and allocation numbers are small. Unless we expect many iterations, it doesn't seem like further optimization is worth much time, effort, and loss of readability.

euler67.go:
package main

import (
    "bufio"
    "flag"
    "fmt"
    "io"
    "os"
    "strconv"
    "time"
)

func parseInts(s string, buf []int) ([]int, error) {
    buf = buf[:0]
    inInt := false
    i := 0
    for j := 0; j <= len(s); j++ {
        if j == len(s) || ('0' > s[j] || s[j] > '9') {
            if inInt {
                n, err := strconv.Atoi(s[i:j])
                if err != nil {
                    return buf[:0], err
                }
                buf = append(buf, n)
            }
            inInt = false
        } else if !inInt {
            inInt = true
            i = j
        }
    }
    return buf, nil
}

func loadTriangle(r io.Reader) ([][]int, error) {
    var ta [][]int
    s := bufio.NewScanner(r)
    for s.Scan() {
        buf := make([]int, 0, len(ta)+1)
        row, err := parseInts(s.Text(), buf)
        if err != nil {
            return nil, err
        }
        if len(row) != len(ta)+1 {
            err := fmt.Errorf(
                "triangle: %d cols for row %d",
                len(row), len(ta)+1,
            )
            return nil, err
        }
        ta = append(ta, row)
    }
    if err := s.Err(); err != nil {
        return nil, err
    }
    return ta, nil
}

func maxPathSum(ta [][]int) int {
    for r := len(ta) - 2; r >= 0; r-- {
        r0, r1 := ta[r], ta[r+1]
        for c0 := range r0 {
            left, right := r1[c0], r1[c0+1]
            if left >= right {
                r0[c0] += left
            } else {
                r0[c0] += right
            }
        }
    }
    if len(ta) == 0 || len(ta[0]) == 0 {
        return 0
    }
    return ta[0][0]
}

func euler67(r io.Reader) (int, error) {
    ta, err := loadTriangle(r)
    if err != nil {
        return 0, err
    }
    return maxPathSum(ta), nil
}

func since(start time.Time) { fmt.Println(time.Since(start)) }

func main() {
    defer since(time.Now())

    flag.Parse()
    // https://projecteuler.net/project/resources/p067_triangle.txt
    // 100 rows 15.2 kB (15,150 bytes)
    filename := `p067_triangle.txt`
    if len(flag.Arg(0)) > 0 {
        filename = flag.Arg(0)
    }
    f, err := os.Open(filename)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    defer f.Close()

    sum, err := euler67(f)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(sum)
}

euler67_test.go:
package main

import (
    "bufio"
    "bytes"
    "io/ioutil"
    "strings"
    "testing"
)

func BenchmarkEuler67(b *testing.B) {
    for N := 0; N < b.N; N++ {
        _, err := euler67(bytes.NewReader(benchData))
        if err != nil {
            b.Fatal(err)
        }
    }
}

func BenchmarkLoad(b *testing.B) {
    for N := 0; N < b.N; N++ {
        _, err := loadTriangle(bytes.NewReader(benchData))
        if err != nil {
            b.Fatal(err)
        }
    }
}

func BenchmarkRow(b *testing.B) {
    var last string
    s := bufio.NewScanner(bytes.NewReader(benchData))
    for s.Scan() {
        last = s.Text()
    }
    if err := s.Err(); err != nil {
        b.Fatal(err)
    }
    cols := len(strings.Fields(last))
    b.ResetTimer()

    for N := 0; N < b.N; N++ {
        buf := make([]int, 0, cols)
        _, err := parseInts(last, buf)
        if err != nil {
            b.Fatal(err)
        }
    }
}

func BenchmarkSum(b *testing.B) {
    for N := 0; N < b.N; N++ {
        b.StopTimer()
        ta, err := loadTriangle(bytes.NewReader(benchData))
        if err != nil {
            b.Fatal(err)
        }
        b.StartTimer()
        _ = maxPathSum(ta)
    }
}

var benchData = func() []byte {
    // https://projecteuler.net/project/resources/p067_triangle.txt
    // 100 rows 15.2 kB (15,150 bytes)
    filename := `p067_triangle.txt`
    data, err := ioutil.ReadFile(filename)
    if err != nil {
        panic(err)
    }
    return data
}()


Finding the nearest defibrillator exercise from CodingGames For my first attempt at Scala I decided to solve a simple exercise on codingame.com.

Based on a string input containing the user location, the defibrillators count and a list of defibrillators entries (id, name, address, phone number, longitude, latitude), I have to print the closest defibrillator from the user. The distance formula is given in the rules.

An example of input:

3,879483
43,608177
3
1;Maison de la Prevention Sante;6 rue Maguelone 340000 Montpellier;;3,87952263361082;43,6071285339217
2;Hotel de Ville;1 place Georges Freche 34267 Montpellier;;3,89652239197876;43,5987299452849
3;Zoo de Lunaret;50 avenue Agropolis 34090 Mtp;;3,87388031141133;43,6395872778854

import math._
import scala.util._

object Solution extends App {
    val playerLongitude = readLine
    val playerLatitude = readLine
    val defibrilatorCount = readInt

    def sanitizeString(string : String) : String = string.replace(',', '.')
    def stringToDouble(string : String) : Double =  sanitizeString(string).toDouble

    val playerLongitudeValue = stringToDouble(playerLongitude)
    val playerLatitudeValue = stringToDouble(playerLatitude)

    def distanceToPlayer(long : Double, lat : Double) :  Double = {
        val x = (long - playerLongitudeValue) * cos((playerLatitudeValue + lat) / 2)
        val y  = lat - playerLatitudeValue
        sqrt(x * x + y * y) * 6371;
    }

    def distanceToPlayer(stringInput : String) : Double = {
        val tokens = sanitizeString(stringInput).split(";")
        distanceToPlayer(tokens(4).toDouble, tokens(5).toDouble)
    }

    var smallestDistance = Double.MaxValue
    var closestDefibName = ""

    for(i <- 0 until defibrilatorCount) {
        val defib = readLine
        val distance = distanceToPlayer(defib)
        if(distance < smallestDistance) {
            smallestDistance = distance
            closestDefibName = defib.split(";")(1)
        }
    }

    println(closestDefibName)
}

The main observation I would make is that this code — particularly the for loop — looks very procedural.  To find the minimum of some collection as determined by some metric, use .minBy[B](f: (A) ⇒ B): A.

For readability, avoid interleaving function definitions with the execution flow.  Put all of the definitions first, if possible.

You potentially split each defibrillator input line twice: once to extract the coordinates, then maybe again if you are interested in extracting the name.  I suggest parsing each line once and constructing a case class object from it.

Your sanitizeString function is rather heavy-handed.  To parse French-formatted decimals, use java.text.NumberFormat.

Note that defibrilatorCount is misspelled.

Suggested solution

import math.{cos, sqrt}
import io.StdIn.{readInt, readLine}
import java.text.NumberFormat
import java.util.Locale

object Solution extends App {
  val frenchFmt = NumberFormat.getInstance(Locale.FRANCE)
  def parseDouble(s: String): Double = frenchFmt.parse(s).doubleValue

  case class GeoObject(long: Double, lat: Double, name: String)

  def parseDefibrillator(line: String): GeoObject = {
    val fields = line.split(";")
    GeoObject(parseDouble(fields(4)), parseDouble(fields(5)), fields(1))
  }

  def distanceMetric(a: GeoObject, b: GeoObject): Double = {
    val x = (a.long - b.long) * cos((a.lat + a.lat) / 2)
    val y = a.lat - b.lat
    6371 * sqrt(x * x + y * y)
    // Note: 6371 * sqrt is optional, if just comparing distances
  }

  val player = GeoObject(parseDouble(readLine), parseDouble(readLine), null)
  val count = readInt
  val nearest = (0 until count).map(_ => parseDefibrillator(readLine))
                               .minBy(distanceMetric(player, _))
  println(nearest.name)
}


Palindromes without character You are given a string of lowercase letters. Your task is to figure out the index of the character whose removal will result in a palindrome. There will always be a valid solution.

Here is my quick solution:

public class Main {

    private static final String AAA = "aaa";

    public static void main(String[] args) {
        System.out.println(palindromeIndex(AAA));
    }

    /**
     * Returns whether a word is palindromic or not
     * @param word
     * @return true if the word is palindromic, false otherwise
     */
    public static boolean isPalindrome(String word) {
        return word.equals(new StringBuffer().append(word).reverse().toString());
    }

    public static int palindromeIndex(String word) {
        if(isPalindrome(word)) {
            return -1;
        }

        StringBuffer sb = new StringBuffer();
        for(int i = 0; i < word.length(); i++) {
            sb.append(word);
            sb.deleteCharAt(i);
            if(isPalindrome(sb.toString())) {
                return i;
            }
            sb.delete(0, word.length());
        }

        /**the question said that there will always be a valid answer
         * so this shouldn't be necessary
         */
        return - 1;
    }
}

Additionally, out of a little dissatisfaction, I came up with an alternative method of figuring out whether a word is a palindrome:

public static boolean isPalindromeAlt(String word) {
    char[] chars = word.toCharArray();
    int wordEnd = word.length() - 1;
    for(int i = 0; i < chars.length; i++) {
        if(chars[i] != chars[wordEnd]) {
            return false;
        }
        wordEnd--;
    }

    return true;
}

But my question is, after actually thinking about it, is the way I have handled the palindrome index part efficient? I don't really like the fact that I am clearing a the stringbuffer after each iteration and using one, I thought that it might be more efficient to use an array or something, could someone please comment on whether my solution is efficient and whether there could be a more efficient way.

I could probably write a more efficient one in Python, I like how Strings are handled as arrays in it.

The time complexity of your solution is O(n ^ 2), where n is the length of the given string. It is possible to do much better. Here is a linear solution:

static int getPalindromeIndex(String word) {
    if (isPalindrome(word))
        return -1;
    // Finds the first pair of characters that don't match.
    int left = 0;
    int right = word.length() - 1;
    while (word.charAt(left) == word.charAt(right)) {
        left++;
        right--;
    }
    // One of them must be removed and it sufficient to remove one of them.
    if (isPalindrome(word.substring(0, left) + word.substring(left + 1)))
        return left;
    return right;
}

The idea is the following:

If it is not a palindrome, there is a pair of positions (i, n - i - 1) such that characters on this positions do not match.
We can remove one of them. It is guaranteed that a solution always exists so we can try to remove the first one and check if a string is a palindrome. If it is the case, then we can just return it. Otherwise we can just return the second one.


Implementing Student Record Java program Is there a better way to implement my code in Java?  I used to code this in C++ and pretty much worked right off the bat. But in Java, it is different, as it won't let me input data on the string array name.  It seems to skip and move on to ask the user for the age rather than starting on name.

package student;
import java.util.*;

public class Student {

public static void main(String[] args) {
   int i, q, z, c, b;
   int x=0;
   String[] name = new String[30];
   int[] age = new int[30];
   String[] course = new String[30];
   String[] year = new String[30];
   String[] section = new String[30];
   int menuChoice;

   Scanner input = new Scanner (System.in);

   start:
   do{

       System.out.println("\t\t\tStudent Record Menu");
       System.out.println("\t\t1. Add Student\t2. View Students\t3. Search Student\t4. Exit");
       System.out.println("Enter a choice: ");
       menuChoice = input.nextInt();

       if (menuChoice==1)
       {
           for (z=x; z<=29; z++)
           {
               System.out.println("Full name:");
               name [z] = input.nextLine();
               System.out.println("Age:");
               age [z] = input.nextInt();
               System.out.println("Course:");
               course [z] = input.next();
               System.out.println("Year:");
               year [z] = input.next();
               System.out.println("Section:");
               section [z] = input.next();
               x++;
               continue start;
           }
       }
               else if (menuChoice==2)
               {
                   for (i=0; i<x; i++)
                   {
                       System.out.println(name[i] + age [i] + course [i] + year [i] + section [i]);
                   }
               }

   } while (menuChoice<4);

}
}

To start with OOP, you should think of the Student class as an Object, a tangible thing like a ball. Say you were given a student. What properties might it have? For example, it may have a name, a gender, a major, a list of courses, etc etc. What could you ask it to do? You could ask it to tell you its name, to tell you whether it is enrolled in a given course or not, whether it shares a course with a given other student or not, etc etc.

In your example, Students appear to have a name, an age, a course, a year, and a section. These are the values that define a student. We can begin a more OOP approach by defining a Student class that adequately stores this information about a student, and allows construction and access to these fields:

public class Student{
    public String name;
    public int age;
    public String course;
    public String year;
    public String section;

    /** Constructs a Student object with the given values */
    public Student(String name, int age, String course, String year, String section){
        this.name = name;
        this.age = age;
        this.course = course;
        this.year = year;
        this.section = section;
    }

}

Now, you may notice that everything in the above example is public. This is (almost certainly) wrong. For example, making the age field public allows anyone to walk up to a student and say "you are now -12 years old". Certainly a bug. Let's revise and only allow get access to be public.

public class Student{
    private String name;
    private int age;
    private String course;
    private String year;
    private String section;

    /** Constructs a Student object with the given values */
    public Student(String name, int age, String course, String year, String section){
        this.name = name;
        this.age = age;
        this.course = course;
        this.year = year;
        this.section = section;
    }

    /** Returns the name of this Student */
    public String getName(){
        return name;
    }

    /** Returns the age of this Student */
    public int getAge(){
        return age;
    }

    /** Returns the course of this Student */
    public String getCourse(){
        return course;
    }

    /** Returns the year of this Student */
    public String getYear(){
        return year;
    }

    /** Returns the section of this Student */
    public String getSection(){
        return section;
    }
}

Now, while a student does have an intrinsic value for their age, you can't access it directly. You can, however, ask the student how old they are with getAge(), which will tell you what you wanted to know.

Finally, let's tackle the main method you were doing in the first place. Now that we have the Student class, we can represent a (name, age, course, year, section) as a single Student object. Instead of having 5 separate arrays, one for each field, we have a single array of Students. Just add this main method to the above Student class:

public static void main(String[] args) {
   int x = 0;
   int menuChoice = -1;
   Student[] students = new Student[30]; //As a note, hard-coding this '30' is a bad idea.
                                         //Probably should be static, final const in class.
   Scanner input = new Scanner (System.in);
   do{
       System.out.println("\t\t\tStudent Record Menu");
       System.out.println("\t\t1. Add Student\t2. View Students\t3. Search Student\t4. Exit");
       System.out.println("Enter a choice: ");
       menuChoice = input.nextInt();

       if (menuChoice==1){
           if(x < 30) { //Able to add new student.
               System.out.println("Full name:");
               String name = input.next();     //This was your error - should be next like the others,
                                               //Not nextLine()
               System.out.println("Age:");
               int age = input.nextInt();
               System.out.println("Course:");
               String course = input.next();
               System.out.println("Year:");
               String year = input.next();
               System.out.println("Section:");
               String section = input.next();

               //Create the new student using the given inputs
               Student s = new Student(name, age, course, year, section);

               //Place in array
               students[x] = s;

               //Increment x for next student placement
               x++;
           } else {  //Not able to add new student
               System.out.println("Can't add new student, students full");
           }
       }
       else if (menuChoice==2) {
           for (int i=0; i<x; i++) {
               Student s = students[i];
               System.out.println(s.getName() + s.getAge() + s.getCourse()
                                      + s.getYear() + s.getSection());
           }
       }
       else if(menuChoice < 1 || menuChoice > 4){
           System.out.println("Unrecognized menu choice; please re-enter");
       }
   } while (menuChoice != 4);

   //Do close your scanners when you're done with them to avoid a resource leak.
   //This is closing System.in (which is bad), but you're code is terminating anyway
   //so its ok
   input.close();
}

Some places you could go from here are using a List (ArrayList will do) instead of an array for student storage in the main method, because arbitrarily limiting the number of students to 30 seems silly. You could also give students more complex behavior (a list of friends, for example) to do some really cool stuff.

Welcome to Java and OOP! If there's any way I can clarify my answer let me know.


Display toast after action I have a React functional component where users can add a movie from a list to their dashboard. I'm displaying a toast (Material UI Snackbar) to show the result. The result can be good: movie has been added, or bad: movie is already added to your watchlist.
const addMovie = (movie: IMovie) => {
  const isMovieDuplicate = !checkForDuplicate(movies, movie);
  if (isMovieDuplicate) {
    movies.push(movie);
    const sortedMovieList = sortMovieList(movies);
    setMovies(sortedMovieList);
    addMovieToLocalStorage(sortedMovieList);
  }
  const message = isMovieDuplicate ?
    movie.original_title + ' has been added to your watchlist.' :
    movie.original_title + ' is already added to your watchlist.';
  const variant = isMovieDuplicate ? {variant: 'success'} : {variant: 'warning'};
  displaySnackbar(message, variant);
};

const displaySnackbar = (message: string, type: any) => {
  enqueueSnackbar(message, type);
};

const sortMovieList = (movies: IMovie[]) => {
  return orderBy(movies, [(movie: IMovie) =>
    returnSortType(movie, sortConfig.selectedSortType)], [sortConfig.orderType ? 'asc' : 'desc'],
  );
};

First I call a function that checks the if the current movie already exists in the movieList array. If so it returns a true. The isMovieDuplicate takes the opposite value, so in case of there being a duplicate it will be false so that the if statement is not executed.
In the if statement I push the movie object in the movies array. Then I use a function to return an sorted array. Users can specify how they want to sort the list based on title, release date.
I use the sortedMovieList value to call my React hook setMovies and put he array in the React state. Then I use that same array and put the data in my local storage. At some point this local storage will be replaced with an actual database.
When the if statement is completed I want to show the result in a snackbar (toast). Based on the isMovieDuplicate variable I create the context for the toast and pass that to the displaySnackbar function which renders it on the page.
This all works, but it doesn't feel "clean". There's quite some code, maybe I could move the code for creating the message into the displaySnackbar method?

Not as clean as it could be, some code duplication, and there's also some state mutation in your addMovie function.
Issues

movies.push(movie); mutates the current state object.
checkForDuplicate isn't a clear name, i.e. checking for duplicates is clear, but what is the return value?
Using isMovieDuplicate as the negation of the result from checkForDuplicate is completely counter-intuitive and confusing.
When adding data to current state and computing the next state, a functional state update should really be used.
Ternary logic could be reduced/simplified.
Handle side-effect of persisting to local storage in the component as an "effect" of updating the movies state.

Suggestions

Use a functional state update and shallow copy of current state in order to not mutate current state and correctly enqueue state updates.
Change checkForDuplicate to checkIsDuplicate to make it more clear the return value is likely a boolean (by isXXX naming convention) and will be true if it is a duplicate.
Remove isMovieDuplicate and use checkIsDuplicate directly in conditional test.
Remove the ternary. Assume duplicate failure, only update if not a duplicate and adding to movie array.
Use an useEffect hook to persist to localStorage (and eventually to DB).

Code
const addMovie = (movie: IMovie) => {
  let message = 'is already added to your watchlist.';
  let variant = 'warning';

  if (!checkIsDuplicate(movies, movie)) {
    setMovies(movies => sortMovieList([...movies, movie]));
    message = 'has been added to your watchlist.';
    variant = 'success';
  }

  displaySnackbar(`${movie.original_title} ${message}`, { variant });
};

useEffect(() => {
  addMovieToLocalStorage(movies);
}, [movies]);


Filtering on an array of object literal properties and their nested array values This is a function which filters on multiple object properties and their corresponding arrays.

I need to match all properties to a pattern and have an array as their property value.
Properties that match that pattern and contain an array get filtered on whether they contain string values.
The values can be in any one of the arrays related to that object.
All objects which have met the criteria and contain the appropriate strings irrespective of the property array they are contained in. With the exception being that all values are in the same object.
The object containing all string values do not get filtered out.

If I match all object properties containing 'ev_' and those properties contain arrays whose values include 'value75' and '12+'

This should be the resulting output

{
  "data": {
    "id": 3,
    "date": "2019-02-03",
    "ev_filter_1": [
      "art",
      "foodie"
    ],
    "ev_filter_2": [
      "value1",
      "value75"
    ],
    "ev_filter_3": [
      "value1",
      "value2"
    ],
    "ev_filter_4": [
      "all",
      "12+"
    ]
  }
}

From this array of objects

const posts = [{
    data: {
      id: 1,
      date: "2019-02-03",
      ev_filter_1: ["art", "foodie"],
      ev_filter_2: ["value1", "value2"],
      ev_filter_3: ["value1", "value2"],
      ev_filter_4: ["all", "12+"]
    }
  },
  {
    data: {
      id: 2,
      date: "",
      ev_filter_1: ["arti", "foodie"],
      ev_filter_2: ["value1", "value2"],
      ev_filter_3: ["value1", "value2"],
      ev_filter_4: ["all", "19+"]
    }
  },
  {
    data: {
      id: 3,
      date: "2019-02-03",
      ev_filter_1: ["art", "foodie"],
      ev_filter_2: ["value1", "value75"],
      ev_filter_3: ["value1", "value2"],
      ev_filter_4: ["all", "12+"]
    }
  }
];

This example is currently working but it is ugly and rigid. The "some" is just used for iteration, which I painfully coerced to return true. There is also a nested if statement (horrid).

const posts = [{
    data: {
      id: 1,
      date: "2019-02-03",
      ev_filter_1: ["art", "foodie"],
      ev_filter_2: ["value1", "value2"],
      ev_filter_3: ["value1", "value2"],
      ev_filter_4: ["all", "12+"]
    }
  },
  {
    data: {
      id: 2,
      date: "",
      ev_filter_1: ["arti", "foodie"],
      ev_filter_2: ["value1", "value2"],
      ev_filter_3: ["value1", "value2"],
      ev_filter_4: ["all", "19+"]
    }
  },
  {
    data: {
      id: 3,
      date: "2019-02-03",
      ev_filter_1: ["art", "foodie"],
      ev_filter_2: ["value1", "value75"],
      ev_filter_3: ["value1", "value2"],
      ev_filter_4: ["all", "12+"]
    }
  }
];

function sift(arrObjLit, pattern, argOne, argTwo) {
  return arrObjLit.filter(function(x, y) {
    return (result = Object.entries(x).some((q, i) => {
      for (let [key, v] of Object.entries(q[1])) {
        if (key.startsWith(pattern) && Array.isArray(v) && v.includes(argOne)) {
          for (let an of Object.values([x.data][0])) {
            if (Array.isArray(an) && an.includes(argTwo)) {
              return true;
            }
          }
        }
      }
    }))
  })
}

console.log(...sift(posts, "ev_", "value75", "12+"));

First, and I can't emphasize this enough, use meaningful variable names. What is argOne? argTwo? The same goes for all of your single letter variable names. They make it harder to look at your code and figure out what's going on.

To help you see where the code you provided might be difficult to understand, I've copied it and commented what came to mind when reading each line, hopefully this can help with writing code that self-documents.

// It looks like this will filter arrObjLit based on pattern.
// But what is argOne, argTwo? Do I need both? What if I include argThree?
function sift(arrObjLit, pattern, argOne, argTwo) {
  // Filtering as expected, but why is the index y needed?
  return arrObjLit.filter(function(x, y) {
    // Where did result come from? Right now it declares a global variable.
    // What is q? Is the index of it, i, needed?
    // Does this need to be wrapped in parenthesis?
    // The above line used function, here there's an arrow function. Why?
    return (result = Object.entries(x).some((q, i) => {
      // Okay, q[1] is the value, but q[0] isn't used, maybe
      // Object.values() should be used above?
      // What happens if q isn't an object?
      for (let [key, v] of Object.entries(q[1])) {
        // This is clear, but what is special about argOne?
        // Why not check argTwo here too?
        if (key.startsWith(pattern) && Array.isArray(v) && v.includes(argOne)) {
          // Why is [x.data][0] used instead of just x.data?
          // x.data == q[1]? What happens if q[1] != x.data?
          for (let an of Object.values([x.data][0])) {
            // Why don't we check that the key for the an array starts
            // with key like we did for argOne?
            if (Array.isArray(an) && an.includes(argTwo)) {
              return true;
            }
          }
        }
      }
    }))
  })
}

In chat, you mentioned that ideally, every array contained within the object should be searched, if it matches the pattern. With this in mind, here is how I would solve your problem. (It might be a good idea to rename pattern to prefix, with pattern I would expect that I can pass in a regular expression)

function sift(entries, pattern, ...filters) {
  return entries.filter(entry => {
    // Copy the filters array into a set
    let toMatch = new Set(filters);

    let searchForMatches = root => {
      for (let [key, value] of Object.entries(root)) {
        if (Array.isArray(value) && key.startsWith(pattern)) {
          // The key for the array matches the pattern, so remove all of these
          // items from the required items to match, if they are found
          value.forEach(item => toMatch.delete(item));
        } else if (typeof value == 'object' && value) {
          // This item isn't null, and is an object.
          // It may contain an array, recurse
          searchForMatches(value);
        }
      }
    };

    searchForMatches(entry);

    // If the set is empty, all matches were found
    // and this object should be included in the result.
    return toMatch.size == 0;
  });
}


SPOJ Emoticon challenge I tried this spoj problem and solved it using Dynamic Programming, but I was getting Time Limit Exceeded. The challenge is to count the number of occurrences of the subsequence "KEK" in each input string.

Some people in comments stated that they have done in O(n) time. I am getting no idea. can someone give me an idea? Here is my dynamic programming approach.

#include<bits/stdc++.h>
using namespace std;
int dp(string s,int in,int len,int pos){
    if(in == len){
        return 0;
    }
    if(pos == 0){
        if(s[in] == 'K'){
            return dp(s,in+1,len,1)+dp(s,in+1,len,0);
        }
        return dp(s,in+1,len,0);
    }
    if(pos == 1){
        if(s[in] == 'E'){
            return dp(s,in+1,len,2)+dp(s,in+1,len,1);
        }
        return dp(s,in+1,len,1);
    }
    if(pos == 2){
        if(s[in] == 'K'){
            return 1+dp(s,in+1,len,2);
        }
        return dp(s,in+1,len,2);
    }
}
int main(){
    int t;
    cin>>t;
    while(t--){
        string s;
        cin>>s;
        int l = s.size();
        int k = dp(s,0,l,0);
        cout<<k<<endl;
    }
    return 0;
}

int dp(string s,int in,int len,int pos){

Where are the comments to explain the meanings of the variables? We're forced to reverse engineer them.

    if(in == len){
        return 0;
    }
    if(pos == 0){
        if(s[in] == 'K'){
            return dp(s,in+1,len,1)+dp(s,in+1,len,0);
        }
        return dp(s,in+1,len,0);
    }
    if(pos == 1){
        if(s[in] == 'E'){
            return dp(s,in+1,len,2)+dp(s,in+1,len,1);
        }
        return dp(s,in+1,len,1);
    }
    if(pos == 2){
        if(s[in] == 'K'){
            return 1+dp(s,in+1,len,2);
        }
        return dp(s,in+1,len,2);
    }
}

This is not dynamic programming. A dynamic programming approach to this problem must work on the following basis:

  Given a structure S which summarises the prefix str, and the next character ch, we need to derive a structure S' which summarises the prefix str + ch.

We have some easy conclusions:

S must provide us with the number of the desired subsequences, because at the end of the DP run we will only have S corresponding to the entire string. So it has a field which we can call kek.
If ch is neither 'E' nor 'K' then S' = S.
If ch != 'K' then S'.kek = S.kek.
If ch == 'K', S must provide us with some way of knowing how much bigger S'.kek should be than S.kek. Obviously that must be the number of KEK subsequences ending with ch, which is the number of KE subsequences in str.

You should be able to carry this reasoning through to fill out the entire list of fields required in S and the logic to update them when reading an 'E' and when reading a 'K'.