Select the Enum to classify a string based on the first word found I have written a small function that will check a string to see if it contains a certain word.

Depending on the word that is found first, the function will return the correct Enum value that needs to be used further down the line of the program.

Would there be a better way to check a string to see if it contains a certain Enum value?
The thing that gives me a little alarm bell in my head is that i am checking the same string 5 times in this function.

Enum

 public enum QuestionIdentifier
{

    Wie,
    Wat,
    Waarom,
    Wanneer,
    Hoe,
    Andere

}

Function

private QuestionIdentifier IdentifyQuestion(string vQuestion)
    {
        //QuestionToLower
        string vQTL = vQuestion.ToLower();

        if (vQTL.Contains("wie"))
        {
            return QuestionIdentifier.Wie;

        }else if (vQTL.Contains("wat"))
        {
            return QuestionIdentifier.Wat;

        }else if (vQTL.Contains("waarom"))
        {
            return QuestionIdentifier.Waarom;

        }else if (vQTL.Contains("wanneer"))
        {
            return QuestionIdentifier.Wanneer;

        }else if (vQTL.Contains("hoe"))
        {
            return QuestionIdentifier.Hoe;

        }else
        {
            return QuestionIdentifier.Andere;
        }

    }

You can utilise the Enum class which can shorten your code significantly and it will be easier for future updates

private static QuestionIdentifier IdentifyQuestion(string vQuestion)
{
    //QuestionToLower
    string vQTL = vQuestion.ToLower();
    string[] enumNames = Enum.GetNames(typeof(QuestionIdentifier));
    foreach (var enumName in enumNames)
    {
        if (vQTL.Contains(enumName.ToLower()))
        {
            return (QuestionIdentifier) Enum.Parse(typeof(QuestionIdentifier), enumName);
        }
    }
    return QuestionIdentifier.Andere;
}

or with LINQ

private static QuestionIdentifier IdentifyQuestion(string vQuestion)
{
    string vQTL = vQuestion.ToLower();
    string[] enumNames = Enum.GetNames(typeof(QuestionIdentifier));
    string enumValueName = enumNames.FirstOrDefault(x => vQTL.Contains(x.ToLower()));
    return enumValueName != null
        ? (QuestionIdentifier) Enum.Parse(typeof(QuestionIdentifier), enumValueName)
        : QuestionIdentifier.Andere;
}

Now you can add as much types as you want in your enum without even touching this method it will still work and find your newly created type.

Update

As @t3chb0t pointed in the comments there is a way to avoid calling .ToLower() on both the input string and the enum value by using the .IndexOf() overload which accepts different comparison types :

private static QuestionIdentifier IdentifyQuestion(string vQuestion)
{
    string[] enumNames = Enum.GetNames(typeof(QuestionIdentifier));
    string enumValueName =
        enumNames.FirstOrDefault(x => vQuestion.IndexOf(x, StringComparison.OrdinalIgnoreCase) >= 0);
    return enumValueName != null
        ? (QuestionIdentifier)Enum.Parse(typeof(QuestionIdentifier), enumValueName)
        : QuestionIdentifier.Andere;
}


Multiplying and adding big numbers represented with strings I got a very unusual problem of adding and multiplying very big numbers (≥ 1e+100). So I've written simple functions that would operate on string representations of numbers, both as an input and an output.

Multiplication:

function multiply(a, b) {
    if ((a | 0) == 0 || (b | 0) == 0) {
        return '0';
    }

    a = a.split('').reverse();
    b = b.split('').reverse();
    var result = [];

    for (var i = 0; a[i] >= 0; i++) {
        for (var j = 0; b[j] >= 0; j++) {
            if (!result[i + j]) {
                result[i + j] = 0;
            }

            result[i + j] += a[i] * b[j];
        }
    }

    for (var i = 0; result[i] >= 0; i++) {
        if (result[i] >= 10) {
            if (!result[i + 1]) {
                result[i + 1] = 0;
            }

            result[i + 1] += parseInt(result[i] / 10);
            result[i] %= 10;
        }
    }

    return result.reverse().join('');
}

Addition:

function add(a, b) {
    if ((a | 0) == 0 && (b | 0) == 0) {
        return '0';
    }

    a = a.split('').reverse();
    b = b.split('').reverse();
    var result = [];

    for (var i = 0; (a[i] >= 0) || (b[i] >= 0); i++) {
        var sum = (parseInt(a[i]) || 0) + (parseInt(b[i]) || 0);

        if (!result[i]) {
            result[i] = 0;
        }

        var next = ((result[i] + sum) / 10) | 0;
        result[i] = (result[i] + sum) % 10;

        if (next) {
            result[i + 1] = next;
        }
    }

    return result.reverse().join('');
}

Are there better ways of doing this job? Some additional edge cases or better loop conditions, maybe?

Use the correct functions to convert to types

I think you are using the bitwise | to convert a string to a number:

if ((a | 0) == 0 || (b | 0) == 0) {
    return '0';
}

I think parseInt can show this intent more clearly:

if (parseInt(a) == 0 || parseInt(b) == 0) {
    return '0';
}


Chrono-like clock with different epoch (from 1.1. 2000) The following will by my simple helper, that should behave just like std::chrono::system_clock but with the epoch being 1.1. 2000, that I can easily cast durations to seconds (or milliseconds or centiseconds) and store them in some records (originated from small devices).

It all started with one constexpr:

/// 1.1.2000 0:00
    static constexpr time_point<system_clock> epoch
      = time_point<system_clock>(seconds(946681200))

Which helped me to get e.g. milliseconds from 1.1.2000 like this:

duration_cast<milliseconds>( system_clock::now() - epoch ).count()

The problem is, that there is no contract / specification what is the epoch, but usually is 1.1. 1970.

(I have once selected 1.1. 2000 because of RTCs in units and designed the firmware to work until 1.1. 2100 when I will probably be dead already or very very old man and won't have to care... record size and amount of data transmitted was more important at the time I have designed the protocol.)

(There are some using std::chrono::system_clock and such inside namespace firda in that header)

#include "basics.hpp"
using namespace firda;

/// Clock with altered epoch
template<class Clock = system_clock
  , typename Clock::rep Epoch = 946681200>
  struct adapted_clock {

    using rep        = typename Clock::rep;
    using period     = typename Clock::period;
    using duration   = typename Clock::duration;
    using time_point = typename firda::time_point<adapted_clock>;

    static constexpr bool is_steady
      = Clock::is_steady;
    static constexpr typename Clock::time_point epoch
      = typename Clock::time_point(seconds(Epoch));

    static time_point now() {
        return time_point(Clock::now() - epoch);
    }

    static time_t to_time_t(const time_point& t) {
        return Clock::to_time_t(epoch + t.time_since_epoch());
    }

    static time_point from_time_t(time_t t) {
        return time_point(Clock::from_time_t(t) - epoch);
    }
};

using y2k_clock = adapted_clock<>;

int main() {
    auto now1 = system_clock::now();
    auto now2 = y2k_clock::now();
    cout << "now1.count: " << duration_cast<seconds>(now1
      .time_since_epoch()).count() << '\n';
    cout << "now2.count: " << duration_cast<seconds>(now2
      .time_since_epoch()).count() << '\n';
    time_t time1 = system_clock::to_time_t(now1);
    time_t time2 = y2k_clock::to_time_t(now2);
    cout << "time1: " << std::ctime(&time1) << '\n';
    cout << "time2: " << std::ctime(&time2) << '\n';
}

Frankly, there isn't much to say about this class since it is simple enough and relies on another simple clock. The type alias, static members and static methods seem to all rely just like they should on the template parameter Clock so there is no problem. I believe that firda::time_point is nothing really from std::chrono::time_point os this shouldn't be a problem either.

The only remarks I have is about the template parameters: I am not sure whether defaulting Clock to std::chrono::system_clock makes sense. Is it really the default clock amongst the three standard clocks? If you think so, why. But frankly, defaulting Epoch has no sense: adapted_clock<> doesn't tell anything about the epoch to the user and you already provide the meaningful type alias y2k_clock (note that using the defaults is always equivalent to using y2k_clock, which makes them useless). You can't default Clock without defaulting Epoch though because you can't change the order of the template parameters.

I think that the best solution would be to leave undefaulted template parameters in adapted_clock but make y2l_clock an alias template with a possibly default Clock template parameter:

template<typename Clock=std::chrono::system_clock>
using y2k_clock = adapted_clock<Clock, 946681200>;

If you don't like this alias template solution, I would still recommend not to default the template parameters of adapted_clock but instead to write this:

using y2k_clock = adapted_clock<std::chrono::system_clock, 946681200>;


Generic Extension Method To Count Descendants I've created a extension method to count every available descendant starting from a single, given key.

The data is a flat list, there's is no hierarchical structure. (Well, not entirely true. There is an hierarchy in the data-model, but not in the list itself)

self-reference      parent-reference   some other members
10045     =>        0                   ..
10046     =>        10045               ..
10044     =>        10063               ..
10061     =>        10044               ..
10063     =>        10046               ..
10064     =>        10046               ..
10021     =>        10061               ..
10028     =>        10000               ..
10030     =>        10000               ..
10031     =>        10000               ..
10000     =>        10000               ..

As an example, counting the descendant of the entry with the self-reference number 10046 would result in 6.

Let's take 10021 as an possible related entry to entry 10046, for example.

  10021 refers to 10061, 10061 refers to 10044, 10044 refers to 10063,
  10063 refers to 10046
  => related!

My extension method is actually split up in two different methods.

public static int CountDescendants<TSource, T>(
    this IEnumerable<TSource> source,
    T startKey,
    Func<TSource, T> selfReference,
    Func<TSource, T> parentReference)
{
    if(source == null) {
        throw new ArgumentNullException("source");
    }

    return Count(source, startKey, selfReference, parentReference);
}

startKey defines our filter, the key we're looking for. In the example above I've used

  10046

selfReference defines a unique key a dataset can be identified with.

parentReference defines a unique key which refers to another dataset.

The actual algorithm looks like this.

private static int Count<TSource, T>(
            IEnumerable<TSource> source,
            T key,
            Func<TSource, T> selfReference,
            Func<TSource, T> parentReference)
{
    // hash-map + .Where is faster than using IEnumerable + .Where
    ReadOnlyDictionary<T, T> consolidatedSource =
        new ReadOnlyDictionary<T, T>(source.ToDictionary(selfReference, parentReference));
    int count = 0;

    // to keep track of every reference which is somehow related to "key"
    HashSet<T> keys = new HashSet<T> {
        key
    };

    // using a stack instead of recurison
    Stack<KeyValuePair<T, T>> stack = new Stack<KeyValuePair<T, T>>();

    // get all first level entries with the given key
    foreach(KeyValuePair<T,T> item in consolidatedSource.Where(x => x.Value.Equals(key))) {
        stack.Push(item);
    }

    // using .Any over .Count because we don't care how many items we have but rather that the stack is not empty
    while (stack.Any()) {
        KeyValuePair<T, T> current = stack.Pop();

        if (keys.Contains(current.Value)) {
            count++;
            keys.Add(current.Key);

            // get all first level entries with the new key
            foreach (KeyValuePair<T, T> item in consolidatedSource.Where(x => x.Value.Equals(current.Key) && !x.Key.Equals(current.Key))){
                stack.Push(item);
            }
        }
    }

    return count;
}

The method can be called like this

this.AllProfiles.CountDescendants(CurrentProfile.Ref, x => x.Ref, x => x.ParentRef);

Is stuff like using a Dictionary to speed up things micro-optimization which I shouldn't worry about? Personally, wherever I can impact performance, I try to get the best out of it. Or at least not make it any worse.

Also, I'm not entirely sure if there are any constraints I can apply to the generic types. I don't think there's one that actually fits and is needed.

Peter already pointed out the main problem with your code: you're doing linear searches when looking for child-nodes, which will blow up to \$O(n^2)\$ for worst-case inputs (very deep trees). Indeed, a lookup table that maps a parent ID to child IDs is much more efficient, and ensures that your method remains \$O(n)\$ for all sorts of input.

Another problem is that your method does not guard against cyclic inputs - it'll run indefinitely. You may want to throw an exception instead, or otherwise prevent the start item from being added to the stack again. It's also a good idea to document this behavior.

Other notes:

There's no need to perform input validation in a separate method here: Count does not use yield, so the input validation will be performed immediately. Note that a modern alternative is to move the yielding part to a local function instead of to a separate method.
When throwing argument-related exceptions, it's better to use nameof(parameterName) instead of "parameterName". This lets the compiler protect you against typos, and works well with code refactoring tools.
I would rename T to TKey, selfReference to keySelector and parentReference to parentKeySelector. I think that better describes their purpose, and it's consistent with other Linq methods.
Use var to cut down on type name repetition: var variableName = new LongTypeName(...); is equivalent to LongTypeName variableName = new LongTypeName(...);.
As Peter already mentioned, there's no need for that keys set: you're only adding items with a known parent ID to the stack, so there's no point in verifying whether their parent ID is known.


Writing a more Pythonic solution to the min triangle path The problem I'm referring to is available here. In quick summary, the problem gives a list of lists such as [[2],[3,4],[6,5,7],[4,1,8,3]] and we must find the minimum path sum: 2 + 3 + 5 + 1 = 11. I took the obvious DP approach:

def minimumTotal(self, triangle):
    # make a memo with n rows and i columns
    # where i is the number of elements in the last row
    n, i = len(triangle), len(triangle[-1])
    memo = [[0] * i for _ in range(n)]
    # add the base case
    memo[0][0] = triangle[0][0]
    for m in range(1, n):
        for k in range(m + 1):
            if (k == 0):
                memo[m][k] = triangle[m][k] + (memo[m-1][0] if m - 1 >= 0 else float("inf"))
            elif (m == k):
                memo[m][k] = triangle[m][k] + (memo[m-1][k-1] if m - 1 >= k - 1 else float("inf"))
            else:
                memo[m][k] = triangle[m][k] + min(memo[m-1][k-1] if m - 1 >= k - 1 else float("inf"), memo[m-1][k] if m - 1 >= k else float("inf"))
    # now loop through the last row and choose the min
    answer = memo[-1][0]
    for l in range(1,i):
        if (memo[-1][l] < answer):
            answer = memo[-1][l]
    return answer

This code is faster than 95% of the other submission times so I'm satisfied with run time. However, what I am interested in is how to (a.) make this code more "Pythonic" and (b.) how to improve the memory usage. This code is more memory efficient than only 8% of submissions so I'm curious on how to better improve this.

Your memo array is pretty inefficient, as it is of size \$n^2\$. There's a lot of wasted cells there, right? So one idea to improve the space complexity would be for each memo[i] to be just the right length. For example, memo[0] would be an array of length 1 instead of length n.. This would reduce space complexity from \$n^2\$ to \$n(n+1)/2\$. Still quadratic, but a small improvement.

But you can see that you don't need to keep all that information. All you care about is the previous row, so there's no need to keep more history than that. You could have an array of length n that represents the previous state and reuse it as you go through the triangle. This would result in a space complexity of \$O(n)\$, a big improvement. Of course, if you could change the input in place you could do a space complexity of \$O(1)\$ - no extra space required :)

I also like to use a bottom up approach to these problems. It's more beautiful, as it makes the solution just bubble up. It would rid you of the last for loop as well, which would make it just a bit faster.

Here's a sketch:

t = [v for v in triangle[-1]]
for i in range(len(triangle) - 2, -1 , -1):
  for j in range(len(triangle[i])):
    t[j] = min(t[j], t[j + 1]) + triangle[i][j]

return t[0]


My own implementation of an hashtable This is my own implementation of an hashtable. I would like to receive some reviews or some feedbacks.

public class HashTableEntry<K,V> {

        private K key;
        private V value;
        private HashTableEntry<K,V> next;

        public HashTableEntry(K key, V value) {
            this.key = key;
            this.value = value;
        }

        public K getKey() {
            return key;
        }

        public void setKey(K key) {
            this.key = key;
        }

        public V getValue() {
            return value;
        }

        public void setValue(V value) {
            this.value = value;
        }

        public HashTableEntry<K, V> getNext() {
            return next;
        }

        public void setNext(HashTableEntry<K, V> next) {
            this.next = next;
        }
}

public class HashTable<K, V> {

    private static int DEFAULT_CAPACITY = 31;

    private HashTableEntry<K,V>[] table;

    @SuppressWarnings("unchecked")
    public HashTable() {
        table = (HashTableEntry<K,V>[])new Object[DEFAULT_CAPACITY];
    }

    @SuppressWarnings("unchecked")
    public HashTable(int capacity) {
        table = (HashTableEntry<K,V>[])new Object[capacity];
    }

    public void put(K key, V value) {

        int index = getIndex(key);

        HashTableEntry<K,V> entry = getLastEntry(table[index]);

        entry.setNext(new HashTableEntry<K,V>(key, value));
    }

    public V get(K key) {

        int index = getIndex(key);

        return findValue(table[index], key);
    }

    private int getIndex(K key) {

        return key.hashCode() % table.length;
    }

    private HashTableEntry<K,V> getLastEntry(HashTableEntry<K,V> entry) {

        if (entry == null) return null;

        HashTableEntry<K,V> currentEntry = entry;

        while(currentEntry.getNext() != null)
            currentEntry = currentEntry.getNext();

        return currentEntry;
    }

    private V findValue(HashTableEntry<K,V> entry, K key) {

        if (entry == null) return null;

        HashTableEntry<K,V> currentEntry = entry;

        while(currentEntry.getNext() != null) {

            if (currentEntry.getKey().equals(key)) {
                return currentEntry.getValue();
            }

            currentEntry = currentEntry.getNext();
        }

        return null;
    }
}

With two tightly related classes like HashTable and HashTableEntry, you should probably nest them inside each other:

public class HashTable<K, V> {
    public static class Entry<K, V> {
        ...
    }
    ...
}

This way, the relationship between the classes is clear.

In your Entry class, you make it mutable. However, why not make it immutable? Immutability wouldn't cost much, but it would make some stuff easier to keep track of, such as providing the user an immutable view of your hashtable (this may be undesired, but it's what I'd do).

public static class Entry<K,V> { // This was nested within HashTable<K, V>
    private final K key;
    private final V value;
    private final Entry<K,V> next;

    public Entry(K key, V value, Entry<K, V> next) {
        this.key = key;
        this.value = value;
        this.next = next; // This could be changed by the user, so we aren't quite immutable yet; we'd have to do a deep copy on `next`
    }

    public K getKey() {
        return key;
    }

    public V getValue() {
        return value;
    }

    public Entry<K, V> getNext() {
        return next;
    }

    public Entry<K, V> withNext(Entry<K, V> newNext) {
        return new Entry<>(key, value, newNext);
    }
}

Interestingly, you are combining table entries with a linked-list. This makes sense from the implementation viewpoint of a hashtable, but for readability's sake, I recommend removing the Entry<K, V> next entirely and simply using an array of LinkedList in your HashTable class. If you don't want to use the built-in LinkedList class, I still feel it is clearer if you separate these two unrelated pieces of the hashtable.

This would require some other changes in the code as well.

In getLastEntry and findValue, you return null when the request cannot be met. However, this is a little bit unclear. I'd recommend instead using Optional<Entry<K, V>> and Optional<V> as your return types so you can just do this:

private Optional<V> findValue(HashTableEntry<K,V> entry, K key) {
    if (entry == null) return Optional.empty();

    HashTableEntry<K,V> currentEntry = entry;

    while(currentEntry.getNext() != null) {
        if (currentEntry.getKey().equals(key)) {
            return Optional.of(currentEntry.getValue());
        }

        currentEntry = currentEntry.getNext();
    }

    return Optional.empty();
}

Then to get the value, you'd call optional.get(), but first you should check optional.isPresent(). This speaks more clearly than a null, so I personally find it easier to follow in this case.


Archiving content in SharePoint libraries I have the following using statement, used specifically for the purpose of archiving content in SharePoint libraries. In this statement my objects are disposed in two places:

finally
{
    web.Dispose();
    site.Dispose();
}

and at the termination of my using statement.

The full using statement is:

using (var site = new SPSite(connectionString))
            {

                SPWeb web = site.OpenWeb(); // open the site
                Console.WriteLine("[{0}] Opened site: {1}", DateTime.Now.ToShortTimeString(), web);
                Console.WriteLine("[{0}] Web Relative URL is: {1}", DateTime.Now.ToShortTimeString(), web.ServerRelativeUrl);

                try
                {
                    // Get your source and destination libraries
                    var source = web.GetList(web.ServerRelativeUrl + @"/Approval%20History%20%20Sales1");
                    var destination = web.GetList(web.ServerRelativeUrl + @"/Approval%20History%20%20Sales");

                    Console.WriteLine("[{0}] Source set to: {1}", DateTime.Now.ToShortTimeString(), source);
                    Console.WriteLine("[{0}] Destination set to: {1}", DateTime.Now.ToShortTimeString(), destination);

                    // Get the collection of items to move, use source.GetItems(SPQuery) if you want a subset
                    SPListItemCollection items = source.Items;

                    // Get the root folder of the destination we'll use this to add the files
                    SPFolder folder = web.GetFolder(destination.RootFolder.Url);

                    Console.WriteLine("[{0}] Moving {1} files from {2} to {3} - please wait...", DateTime.Now.ToShortTimeString(),
                                      items.Count, source, destination);

                    // Now to move the files and the metadata
                    foreach (SPListItem item in items)
                    {
                        //Get the file associated with the item
                        SPFile file = item.File;

                        // Create a new file in the destination library with the same properties
                        SPFile newFile = folder.Files.Add(folder.Url + "/" + file.Name, file.OpenBinary(), file.Properties, true);

                        // Optionally copy across the created/modified metadata
                        SPListItem newItem = newFile.Item;
                        newItem["Editor"] = item["Editor"];
                        newItem["Modified"] = item["Modified"];
                        newItem["Modified By"] = item["Modified By"];
                        newItem["Author"] = item["Author"];
                        newItem["Created"] = item["Created"];
                        newItem["Created By"] = item["Created By"];

                        // UpdateOverwriteVersion() will preserve the metadata added above.
                        newItem.UpdateOverwriteVersion();

                        // Delete the original version of the file
                        // todo: make local backup before deleting?
                        file.Delete();
                        fileCount++;
                    }

                    Console.WriteLine("[{0}] Completed moving {1} files to {2}", DateTime.Now.ToShortTimeString(), fileCount,
                                      destination);
                }
                catch (System.IO.FileNotFoundException)
                {
                    Console.WriteLine("[{0}] Unable to set a location. Please check that paths for source and destination libraries are correct and relative to the site collection.", DateTime.Now.ToShortTimeString());
                }
                finally
                {
                    web.Dispose();
                    site.Dispose();
                }
            }

Is my finally statement here redundant? Is there any reason that even if potentially it is redundant that I should retain it anyway?

Well, one is redundant, Dispose() on the site object is done for you automatically in the context of using. To be idiomatic in C#, you should do the following:

using (var site = new SPSite(connectionString))
using (SPWeb web = site.OpenWeb()) // open the site
{
        try
        {
            Console.WriteLine("[{0}] Opened site: {1}", DateTime.Now.ToShortTimeString(), web);
            Console.WriteLine("[{0}] Web Relative URL is: {1}", DateTime.Now.ToShortTimeString(), web.ServerRelativeUrl);

            // Get your source and destination libraries
            var source = web.GetList(web.ServerRelativeUrl + @"/Approval%20History%20%20Sales1");
            var destination = web.GetList(web.ServerRelativeUrl + @"/Approval%20History%20%20Sales");

            Console.WriteLine("[{0}] Source set to: {1}", DateTime.Now.ToShortTimeString(), source);
            Console.WriteLine("[{0}] Destination set to: {1}", DateTime.Now.ToShortTimeString(), destination);

            // Get the collection of items to move, use source.GetItems(SPQuery) if you want a subset
            SPListItemCollection items = source.Items;

            // Get the root folder of the destination we'll use this to add the files
            SPFolder folder = web.GetFolder(destination.RootFolder.Url);

            Console.WriteLine("[{0}] Moving {1} files from {2} to {3} - please wait...", DateTime.Now.ToShortTimeString(),
                              items.Count, source, destination);

            // Now to move the files and the metadata
            foreach (SPListItem item in items)
            {
                //Get the file associated with the item
                SPFile file = item.File;

                // Create a new file in the destination library with the same properties
                SPFile newFile = folder.Files.Add(folder.Url + "/" + file.Name, file.OpenBinary(), file.Properties, true);

                // Optionally copy across the created/modified metadata
                SPListItem newItem = newFile.Item;
                newItem["Editor"] = item["Editor"];
                newItem["Modified"] = item["Modified"];
                newItem["Modified By"] = item["Modified By"];
                newItem["Author"] = item["Author"];
                newItem["Created"] = item["Created"];
                newItem["Created By"] = item["Created By"];

                // UpdateOverwriteVersion() will preserve the metadata added above.
                newItem.UpdateOverwriteVersion();

                // Delete the original version of the file
                // todo: make local backup before deleting?
                file.Delete();
                fileCount++;
            }

            Console.WriteLine("[{0}] Completed moving {1} files to {2}", DateTime.Now.ToShortTimeString(), fileCount,
                              destination);
        }
        catch (System.IO.FileNotFoundException)
        {
            Console.WriteLine("[{0}] Unable to set a location. Please check that paths for source and destination libraries are correct and relative to the site collection.", DateTime.Now.ToShortTimeString());
        }
}


Find all files that match a filename pattern in a folder tree I am an old python newbie here, migrating from fortran77 ;)
Still unsure of the python terminology at the moment :(

The python3 script below fetches a large list of matching files in a folder tree, in the format:

[ fully_qualified_folder_name, [list of matching files in this folder] ]

Being only a python newbie, there HAS to be a better way ?

I hope to use the code to generate a nightly json file underpinning a chromecasting web server on a Raspberry Pi 4, based on Google's example chrome (web page) "sender".
The web page will eventually use javascript on the client-side for the user to choose a file from a list and "cast" it from the Pi4's apache2 web server over to the chromecast device.
So, I thought, why not give try Python a try ?

Suggestions would be appreciated.

import os
import fnmatch
def yield_files_with_extensions(folder_path, file_match):
    for root, dirs, files in os.walk(folder_path):
        for file in files:
            if fnmatch.fnmatch(file.lower(),file_match.lower()):
                #yield os.path.join(root, file)
                yield file
        break  # without this line it traverses the subfolders too
def yield_files_in_subfolders(folder_path, file_match):
    for root, dirs, files in os.walk(folder_path):
        for d in dirs:
            subfolder = os.path.join(root, d)
            mp4_files = [subfolder, [f for f in yield_files_with_extensions(subfolder, file_match)]]
            yield mp4_files
#
the_path = r'/mnt/mp4library/mp4library'
# retieve the topmost folder's files
mp4_files1 = [[ the_path, [f for f in yield_files_with_extensions(the_path, "*.mp4")] ]]
mp4_files2 = [f for f in yield_files_in_subfolders(the_path, "*.mp4")]
mp4_files2 = mp4_files1 + mp4_files2
crecords=0
cfiles=0
for a in mp4_files2:
    crecords=crecords+1
    print ("-----record "+str(crecords)+"---" + a[0] + " ... files=" + str(len(a[1])))
    #print (a[0])
    #print (a[1])
    c=0
    for b in a[1]:
        c=c+1
        print ("mp4 File "+str(c)+"---" + b)
    cfiles=cfiles+c
print ('Count of files: ' + str(cfiles))

As you tagged this Python 3.x, I'd suggest using pathlib.Path.rglob() and collections.defaultdict.  Also, check out f-strings.

import collections
import pathlib

mp4s = collections.defaultdict(list)

root = pathlib.Path(r'/mnt/mp4library/mp4library')

file_count = 0

for filepath in root.rglob('*.mp4'):
    #filepath is relative to root. Uncomment the next line for absolute paths
    #filepath = filepath.resolve()

    mp4s[filepath.parent].append(filepath.name)

    file_count += 1

for record_number, (folder, filenames) in enumerate(sorted(mp4s.items())):

    print(f"-----record {record_number}---{folder} ... files={len(filenames)}")

    for c,filename in enumerate(filenames):
        print(f"mp4 File {c}---filename")

print('Count of files: {filecount}')


Basic, single-threaded implementation of SynchronizationContext I am trying to write a SynchronizationContext in C# that represents a message queue, to be pumped from a main loop.

Edit: I see that I have forgotten to say - I need the message loop executions to be done a single, pre-determined thread. (In my case I want to push updates to an OpenGL context.)

Here is the class I have written:

public class SingleThreadedSynchronizationContext : SynchronizationContext
{
    private sealed class WorkItem
    {
        private readonly SendOrPostCallback _callback;
        private readonly object _state;
        private readonly ManualResetEventSlim _reset;

        public WorkItem (SendOrPostCallback callback, object state, ManualResetEventSlim reset)
        {
            if (callback == null)
                throw new ArgumentNullException ("callback");

            _callback = callback;
            _state = state;
            _reset = reset;
        }

        public void Execute ()
        {
            _callback (_state);
            if (_reset != null) {
                _reset.Set ();
            }
        }
    }

    private readonly ConcurrentQueue<WorkItem> _workItems = new ConcurrentQueue<WorkItem> ();
    private readonly Thread _executingThread;

    public SingleThreadedSynchronizationContext (Thread executingThread)
    {
        if (executingThread == null)
            throw new ArgumentNullException ("executingThread");
        _executingThread = executingThread;
    }

    internal bool HasWorkItems {
        get {
            return !_workItems.IsEmpty;
        }
    }

    private WorkItem ExecuteAndReturnNextWorkItem ()
    {
        WorkItem currentItem;
        if (_workItems.TryDequeue (out currentItem)) {
            currentItem.Execute ();
        }
        return currentItem;
    }

    private void ExecuteNextWorkItem ()
    {
        ExecuteAndReturnNextWorkItem ();
    }

    public override void Post (SendOrPostCallback d, object state)
    {
        _workItems.Enqueue (new WorkItem (d, state, null));
    }

    public override void Send (SendOrPostCallback d, object state)
    {
        if (Thread.CurrentThread == _executingThread) {
            WorkItem requestedWorkItem = new WorkItem (d, state, null);
            _workItems.Enqueue (requestedWorkItem);
            WorkItem executedWorkItem = null;
            do {
                executedWorkItem = ExecuteAndReturnNextWorkItem ();
            } while (executedWorkItem != null && executedWorkItem != requestedWorkItem);
        } else {
            using (var reset = new ManualResetEventSlim ()) {
                _workItems.Enqueue (new WorkItem (d, state, reset));
                reset.Wait ();
            }
        }
    }
}

And here is a (everything additional removed) example of my main loop:

class MainClass
{
    public static void Main (string[] args)
    {
        var synchronizationContext = new CustomSynchronizationContext ();
        SynchronizationContext.SetSynchronizationContext (synchronizationContext);

        while (true) {
            // Do something else in my loop;

            while (synchronizationContext.HasWorkItems) {
                synchronizationContext.ExecuteNextWorkItem ();
            }
        }
    }

It is my first attempt at doing something on that level in the CLR concerning threading, so I wonder whether I have made any mistake in writing it, or done some other kind of mispractice (e.g. wrong class, etc.).

You don't need to implement Dispose pattern if you have no unmanaged resources to dispose. Unless you get paid per line of code, that is :)
You should not dispose objects you did not create (unless you implement a wrapper). In your case, you should not dispose reset event in WorkItem class, since you did not create it there. Instead, you should use

using(var reset = new ManualResetEventSlim())
{
    _workItems.Enqueue (new WorkItem (d, state, reset));
    ...
}

this code:

while (true)
{
    synchronizationContext.Send(_ => Console.WriteLine("Hello!"), null)

    while (synchronizationContext.HasWorkItems) {
        synchronizationContext.ExecuteWorkItem ();
    }
}

will hang your application. You should either remove resetEvent from Send method and simply execute all queued WorkItems there (bad approach) or move this loop:

    while (synchronizationContext.HasWorkItems) {
        synchronizationContext.ExecuteWorkItem ();
    }

to sepatrate thread (better approach). Might look like:

public sealed class CustomSynchronizationContext : SynchronizationContext, IDisposable
{
     public CustomSynchronizationContext()
     {
         _thread = new Thread(() =>
               {
                   while(true)
                   {
                       _itemQueuedEvent.WaitOne();
                       if (_disposed) return;
                       ExecuteWorkItem();
                   }
               });
         _thread.Start();
     }

 ........
}

You will probably encounter some synchronization issues, but nothing that can not be solved.


Shuffling an array of cards I have a deck of flashcards that I want to shuffle.  Here's the method I'm using to shuffle them:

func (deck *Deck) Shuffle() {
        rand.Seed(time.Now().UnixNano())
        randomIndexes := rand.Perm(len(deck.Cards))

        shuffledCards := make([]Card, len(deck.Cards))
        for i := 0; i < len(deck.Cards); i++ {
                shuffledCards[i] = deck.Cards[randomIndexes[i]]
        }

        deck.Cards = shuffledCards
}

Is this an efficient way of doing it, or is there a better way?

I would recommend something closer to a Knuth shuffle, where you swap items in place in the array, rather then allocating a new one.

This is untested and is based on the Fisher-Yates shuffle:

func (deck *Deck) Shuffle() {
    rand.Seed(time.Now().UnixNano())

    for i := len(deck.Cards)-1; i > 0; i-- {
        j := rand.Intn(i+1) // i+1 rather than i; the upper bound is not inclusive
        deck.Cards[i], deck.Cards[j] = deck.Cards[j], deck.Cards[i]
    }
}

Unrelated to shuffling, you should also check out bucketed flashcards.  Consider the Leitner system.


Error Handling When Using Dictionary I have set up a Dictionary that calls on a class to fill a DataGridView via SQL statements.  The problem is in two (out of 5) instances the value passed HAS to be an integer, but the value comes from a textbox so it is being passed as string.  Before I set these Dictionaries up I was just using a switch statement that can be seen here.  In my switch statement I simply added:

int n;
bool isNumber = int.TryParse(textBox.Text, out n);

if (!isNumber)
{
  MessageBox.Show("Input must be an integer");
  break;
}

to the two cases that needed the value to be an integer type.

Any idea on how I can add error handling to this? Preferably a message box that just says "value needs to be an integer".

    private void findScriptsQueryButton_Click(object sender, EventArgs e)
    {
        SqlConnection connection = new SqlConnection();
        string value = this.valueTextBox.Text;
        connection.ConnectionString = GetSqlConnection[serverComboBox.SelectedItem.ToString()];
        findScriptsDataGrid.DataSource = GetDataSource[findByComboBox.SelectedItem.ToString()](connection, value).Tables[0];
    }
    private static TableAdapters.FindScript dataSource = new TableAdapters.FindScript();
    private static readonly Dictionary<String, Func<SqlConnection, String, DataSet>> GetDataSource = new Dictionary<String, Func<SqlConnection, String, DataSet>>()
    {
        {"Target VDN", dataSource.FillByTargetVDN},
        {"Skill Group", dataSource.FillBySkillGroup},
        {"Translation Route Pool", dataSource.FillByTranslationRoutePool},
        {"Name", dataSource.FillByName},
        {"Label", dataSource.FillByLabel}
    };

    private static readonly Dictionary<String, String> GetSqlConnection = new Dictionary<String, String>()
    {
        {"SERVER01", ConfigurationManager.ConnectionStrings["csS01"].ConnectionString},
        {"SERVER02", ConfigurationManager.ConnectionStrings["csS02"].ConnectionString}
    };

In your data structure include a validator (this should probably be a Predicate<String>); most of the predicates will just return true, but the two that require integers will have a nontrivial validator that checks for integer.

So I guess the type should be Dictionary<String, Tuple<Func<SqlConnection, String, DataSet>>,Predicate<String>>


Code reuse while keeping meaning clear and avoiding unforseen consequences I like to reuse code as much as possible and keeps things DRY. But I'm afraid that sometimes it creates too many layers of indirection. It then becomes a chore to find where code exists that is relevant to the problem at hand. I'll give a couple of examples.

In the first example, I'm using inheritance on classes that are not directly related, but one class happens to need a lot of functions from the other class.

Here's a portion of the base class:

class GetTimecard < Interactor
  def date_range_start
    if @request.params[:date_range_start]
      Date.parse(@request.params[:date_range_start])
    else
      pay_period.last_starts_on
    end
  end

  def date_range_end
    if @request.params[:date_range_end]
      Date.parse(@request.params[:date_range_end]) + 1.day
    else
      pay_period.last_ends_on
    end
  end
end

The inherited class is largely the same as base class but does not need a date range. It is only interested in a single day.

class GetTimecardForShift < GetTimecard
  def call
    @request.params[:date_range_start] = @request.params[:date]
    @request.params[:date_range_end] = @request.params[:date]
  end
end

So in the call method you have a strange assignment of variables that only makes sense if you have a look at the base class.

In the second example, I use modules to share common functionality between many classes.

There is a module like this:

module Timeable
  def timesheet_entries_for(shift)
    TimesheetEntryRepository.where(account_and_user_ids.merge(assigned_shift_id: shift.id))
  end
end

This function calls a method, account_and_user_ids which is not defined in the module itself. Instead, it's defined in each class that includes the module. But each class may define it differently. For example:

class RecordTime < Interactor
  include Timeable

  def account_and_user_ids
    { account_id: @request.account_id, user_id: @request.current_user.id }
  end
end

class GetTimecard < Interactor
  include Timeable

  def account_and_user_ids
    @account_and_user_ids ||= { account_id: @request.account_id, user_id: get_user_ids }
  end

  def get_user_ids
    if @request.params[:id]
      @request.params[:id]
    else
      if @request.params[:departments]
        UserRepository.by_account(@request.account_id).where(department_id: @request.params[:departments]).map(&:id) & managed_users.map(&:id)
      else
        managed_users.map(&:id)
      end
    end
  end
end

So it quickly becomes confusing trying to figure out which method is defined where and how changing one method will affect other code.

Are there standard ways of making these dependencies explicit? Hopefully there's a naming convention or a better way to names things. I'm not a fan of too many comments in code because they have a tendency to tell less than the code itself and even to be flat out wrong.

These dependencies can't be fixed by naming. They are a symptom of some fundamental design problems. If you're having trouble finding methods, that's
a symptom of an overly coupled design.

You have a class names that really sound like methods in a completely different class.

Without knowing more about the application it's hard to suggest where to go. If you're working in Ruby I highly recommend this book for thinking about class design and re-factoring.

http://www.poodr.com/

You might also look into various code metric tools to help you identify some design problems. There are lot's of these available for Ruby.

https://www.ruby-toolbox.com/categories/code_metrics

These can seem quite pointless and arbitrary at first, but using SOLID object design principals has been shown to be effective in the long run.

http://en.wikipedia.org/wiki/SOLID_(object-oriented_design)


implementation of atoi in scala I've implemented atoi function as described in leetcode problem string-to-integer-atoi.

I am looking for some feedback on writing this in a more scala native way and being able to include the requirement that:

  If the numerical value is out of the range of representable values,
  INT_MAX (231 − 1) or INT_MIN (−231) is returned.

  object Solution {
    def myAtoi(str: String): Int = {

      val charMap = Map('1'->1, '2'->2, '3'->3, '4'->4, '5'->5, '6'->6, '7'->7, '8'->8, '9'->9, '0'-> 0)

      val keySet = charMap.keySet
      var num = 0
      var firstChar = -1
      var negFlag = false
      var zeroFlag = false

      var cleanStr = str.reverse.dropWhile(c => !charMap.contains(c) || c == '-' || c == ' ').trim

      for((c, i) <- cleanStr.view.zipWithIndex) {
        var summand = c match {
          case c if keySet.contains(c) => {
            if(firstChar == -1) firstChar = i
            charMap(c) * math.pow(10, i - firstChar).toInt
          }
          case '-' if(firstChar != -1) =>  {
            negFlag = true
            0
          }
          case _ => {
            zeroFlag = true
            0
          }
        }
        num += summand
      }
      if(zeroFlag) 0 else if(negFlag) num * -1 else num
    }
  }

A few observations about your code.

incomplete - As you've noted, it doesn't handle Int overflow. It also doesn't handle the optional leading + character.

digit characters - The variables charMap and keyMap are unnecessary. The .isDigit and .asDigit methods in the standard library provide the same functionality.

Boolean logic - Consider this Boolean test: !charMap.contains(c) || c == '-' || c == ' ' It translates to "if c is not a digit OR if c is a dash OR if c is a space." But if c is a dash or space then it is already not a digit. The dash and space tests serve no purpose because no dash or space character would get beyond the digit test.

unneeded vars - There's no reason to make cleanStr or summand a var. Their initial values are never changed. In fact, you could just drop summand altogether and increment num directly.

num += (c match { ...//rest of code block

excess braces - In a match{...} every case statement ends the previous code block and starts a new one. Therefore wrapping the code block in { braces } is redundant and just adds visual clutter. In other words, this...

case x =>
  //code
  //here

...is cleaner than this...

case x => {
  //code
  //here
}

style - The use of so much mutation (i.e. vars instead of vals) makes it pretty clear that you are using the Scala language to write C code. This will change as you become more familiar with the Scala standard library, the tenants of Functional Programming, and the study of other Scala code.

Here, for example, is a Scala solution for the atoi challenge.

val numRE = "\\s*([+-]?)0*(\\d+).*".r

def myAtoi(str :String) :Int = str match {
  case numRE(signStr, numStr) =>
    val sign    = if (Option(signStr).contains("-")) -1 else 1
    val bailOut = if (sign < 0) Int.MinValue else Int.MaxValue
    if (numStr.length > 10) bailOut                      //too many digits
    else if (numStr.length < 10 || numStr < "2147483648")
      sign * numStr.foldLeft(0)(_*10 + _.asDigit)        //calculate return Int
    else bailOut
  case _ => 0  //input doesn't match the pattern
}


Hexadecimal converter using a switch statement I need some help improving my code and wondering if I there is a better way to do the following:

StringBuilder buf = new StringBuilder(node.length() + 8);
        for (int i=0, n=node.length(); i<n; i++) {
            char c = node.charAt(i);
            String hexadecimal = "";
            switch (c) {
                case '"': hexadecimal = "\\22"; break;
                case '&': hexadecimal = "\\26"; break;
                case '\'': hexadecimal = "\\27"; break;
                case '/':  hexadecimal = "\\2f"; break;
                case ':': hexadecimal = "\\3a"; break;
                case '<': hexadecimal = "\\3c"; break;
                case '>': hexadecimal = "\\3e"; break;
                case '@': hexadecimal ="\\40"; break;
                case '\\':hexadecimal = "\\5c"; break;
                default: {
                    if (Character.isWhitespace(c)) {
                        hexadecimal = "\\20";
                    }else if(c > 127){
                        hexadecimal = "\\" + Integer.toHexString(c) + "\\";
                    }
                    else {
                        hexadecimal = String.valueOf(c);
                    }
                }
            }
            buf.append(hexadecimal);
        }
        return buf.toString();

I don't like how there is a switch statement which just sets a variable (hexadecimal) based on what the case is. I feel there is a better way to do this, such as by using a hash-map. I need the code to be flexible so that adding new characters and setting the hexadecimal is easy.

A hash map is often a good idea for avoiding switch statements. It can be used in combination with the Strategy and Abstract Factory patterns, such as in these examples.

However, the Strategy Pattern is overkill for your problem. A simple hashmap would look like this

Map<Character, String> hexMap = new HashMap<Character, String>();
hexMap.put('\"', "\\22");
hexMap.put('\n', "\\20");
// puts more stuff in map...

 for (int i=0, n=node.length(); i<n; i++) {
     char c = node.charAt(i);
     buf.append(hexMap.get(c));
 }

But a quick googling of "java character to hexadecimal" produces these results. My favorite is

StringBuilder buf = new StringBuilder(node.length() + 8);

    for (int i = 0; i < node.length(); ++i) {
        char ch = node.charAt(i);
        buf.append(String.format("\\%1$x", (ch & 0xFFFF)));
    }

return buf.toString();

EDIT: Forgot to say, don't use magic numbers. I have no clue what the frack that 8 is doing in your program nor do I have to patience to try to deduce.


Data selection and filtering Here i have javascript code where is 'animals' object and 'people' array of objects. User selects the data from select box and accordingly table fills with selected data(which is array of objects) and when user wants to reset everything to the state before selected data, user just clicks 'backtozero' button.
the code is a little complicated to understand for beginners specially two 'for loops'(i know foreach but maybe someting else ?) and also information which is inside of 'addEventListeners function' any suggestion for making whole thing easier to read for beginners ? feel free to make changes :

let animals
let animalCols = ['Animal', 'Animal 2']
let peopleCols = ['Person', 'Person 2']

function myFunction() {
  paivitys(animals, animalCols)
}

function paivitys(dataa, arvvoja) {
  console.log(dataa);
  //----
  if (dataa.hasOwnProperty("animal")) {
    document.getElementById("1name").innerHTML = dataa.animal;
  } else {
    document.getElementById("1name").innerHTML = dataa.person;
  }
  //----
  if (dataa.hasOwnProperty("animal2")) {
    document.getElementById("2name").innerHTML = dataa.animal2;
  } else {
    document.getElementById("2name").innerHTML = dataa.person2;

  }

  document.getElementById("1name1").innerHTML = arvvoja[0];
  document.getElementById("2name1").innerHTML = arvvoja[1];

  //-----
  document.getElementById("id").innerHTML = dataa.id;
}

function paivitaselekt(araytassa, arvvoja) {

var i, j;
for (i = 0; i < araytassa.length; i++) {
    var ssellecct = document.getElementById("Select");
    var oppttion = document.createElement("option");
    for (j = 0; j < arvvoja.length; j++) {
       oppttion.textContent += araytassa[i][arvvoja[j]] + " ";
    }
    ssellecct.appendChild(oppttion);
}
}

animals = {
  "animal": "tiger",
  "animal2": "lion",
  "id": "54321",
  "dole": {
    "Key": "fhd699f"
  }
}

paivitys(animals, animalCols);
let infoarray;

people = [{
    "person": "kaka",
    "person2": "julle",
    "id": "9874",
  },
  {
    "person": "Ronaldo",
    "person2": "jussi",
    "id": "65555",
  }
]
infoarray = people;
paivitaselekt(infoarray, ["person", "id"]);

document.getElementById("Select").addEventListener("change", function(event) {
  const chosenid = event.target.value.split(" ")[1];
  const choseninfo = infoarray.filter((dataa) => dataa.id === chosenid)[0];
  paivitys(choseninfo, peopleCols);
});
<!DOCTYPE html>

    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width" />
    <link
      rel="stylesheet"
      href="https://maxcdn.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css"
    />
    <link
      rel="stylesheet"
      href="https://use.fontawesome.com/releases/v5.6.3/css/all.css"
      integrity="sha384UHRtZLI+pbxtHCWp1t77Bi1L4ZtiqrqD80Kn4Z8NTSRyMA2Fd33n5dQ8lWUE00s/"
      crossorigin="anonymous"
    />

        Animal
        Animal
        id

  <select id="Select" ></select>
  <button onclick="myFunction()">backtozero</button>

Indent your code!
Naming convention

Only use English

paivitys -> update -> though really this should be with a subject, so updateObjekt?
dataa -> data
arvvoja -> values
paivitaselekt -> updateSelection

Dont mispel

ssellecct -> selectElement
oppttion -> option

Use lowerCamelCase

infoarray -> infoArray
chosenid -> chosenId
choseninfo -> chosenInfo

myFunction -> this is an unfortunate name, it does not say anything about its functionality
The names of your HTML elements id's can be improved

Dont mix var and const/let
Don't console.log() in production code
Comments should improve understanding, //------ does not do that, just use newlines/whitespace
Dont assign onclick events in html with onclick, use addEventLister()
It seems people is a global variable, don't create global variables
It seems you create people and infoarray, then assign people to infoarray, and then never use people again. It does not seem to make sense


Clumsy PHP5 code reading from an SQLite3 database Some time ago I wrote myself a simple blog. The index page is generated
with PHP5, using an SQLite3 database via the variable id handed to it via
the URL.

The SQLite database contains a table called pages. The id refers of course
to the id intergers in this table. Other columns are title, body, time and
timezone.

The title and body are text as you'd expect, the timezone is usually
"Asia/Tokyo", and the time is seconds from the epoch (I just use date +%s,
when I add an entry, nothing fancy).

I use the following code to get these variables from a table, and fill in
a generic page with them.

<?php
    // Open the database.
    $db = new PDO("sqlite:pagesdatabase.db");
    // Is id set? If not get id of last page and put it in $id.
    if (!isset($_GET["id"])) {
        foreach ($db->query("select max(id) from pages") as $lastid) {
            $id = $lastid[0];
        }
    }
    // If id is set, get the value and put it in $id.
    else {
        $id = $_GET['id'];
    }
    // Using the id number, grab the title, body, time and timezone of post.
    foreach ($db->query("select * from pages where id = $id") as $page) {
        $title = $page['title'];
        $body = $page['body'];
        $time = $page['time'];
        date_default_timezone_set($page['timezone']);
    }
?>

This code works, but seems rather clumsy to me. For example, foreach
appears twice, but shouldn't be necessary. To be honest, it's been a while
since I wrote this code, and for some reason I had trouble using sqlite_open
and other more obvious functions. Can someone help me write the above in a more
terse way?

You are right, the foreaches can be eliminated.  The first foreach which grabs the highest id, can be eliminated by taking advantage of the fact that the query will only ever return a single cell and using the PDOStatement::fetchColumn() method.  Since page ids are unique, the second foreach can be removed because the second query will only ever return a single row.

An additional problem is that you are not sanitizing user input in your SQL.  This is easy to fix with PDO by using a parameterized statement:

<?php
// Open the database.
$db = new PDO("sqlite:pagesdatabase.db");

// Is id set? If not get id of last page and put it in $id.
if (!isset($_GET["id"])) {
    $id = $db->query("select max(id) from pages")->fetchColumn();
}
// If id is set, get the value and put it in $id.
else {
    $id = $_GET['id'];
}

// You may consider doing an additional check here in case there are no pages, in which
// case id will be null
if ($id === null) {
    // ...
    exit;
}

// Using the id number, grab the title, body, time and timezone of post.
$page = $db->prepare("select * from pages where id = :id")
           ->execute(array('id' => $id))
           ->fetch();

// A check for an invalid id should be made here
if ($page === null) {
    // ...
    exit;
}

$title = $page['title'];
$body = $page['body'];
$time = $page['time'];
date_default_timezone_set($page['timezone']);
?>

I have only quickly prepared this answer from experience and have not actually run the code above.  It should be enough to give you an idea but may have syntax errors.  If it does, comment on the post and I will fix it.  Also, if reducing this code to the fewest lines possible is your goal, setting $id can be accomplished with a ternary and the actual retrieval of the page along with the error checks can be encapsulated in a function or a class.

<?php
// Open the database.
$db = new PDO("sqlite:pagesdatabase.db");

// Determine if a page id is provided, if not use the most recent page
$id = isset($_GET['id'])
  ? $_GET['id']
  : $db->query("select max(id) from pages")->fetchColumn();

try {
    $page = Page::fetch($id, $db);

    $title = $page->getTitle();
    $body = $page->getBody();
    $time = $page->getTime();
    date_default_timezone_set($page->getTimezone());

} catch (Exception $e) {
    // This exception should be thrown by Page::fetch if no pages exist or an invalid
    // id is given
    // ...
    exit;
}
?>

If this is preferred I will leave the class definition to you.


Converting audio to different file formats I have this code that converts audio to different file formats:

import java.io.File;
import it.sauronsoftware.jave.AudioAttributes;
import it.sauronsoftware.jave.Encoder;
import it.sauronsoftware.jave.EncoderException;
import it.sauronsoftware.jave.EncodingAttributes;
import it.sauronsoftware.jave.InputFormatException;

/*  required jars(jave-1.0.2.jar)
 *  Handles Encoding and Decoding
 *  Audio to different file formats
 */
public class AudioEncoderDecoder {

    private static final Integer bitrate = 256000;//Minimal bitrate only
    private static final Integer channels = 2; //2 for stereo, 1 for mono
    private static final Integer samplingRate = 44100;//For good quality.

    /* Data structures for the audio
     *  and Encoding attributes
     */
    private AudioAttributes audioAttr = new AudioAttributes();
    private EncodingAttributes encoAttrs = new EncodingAttributes();
    private Encoder encoder = new Encoder();

    /*
     * File formats that will be converted
     * Please Don't change!
     */
    private String oggFormat = "ogg";
    private String mp3Format = "mp3";
    private String wavFormat = "wav";

    /*
     * Codecs to be used
     */
    private String oggCodec = "vorbis";

    /* Set the default attributes
     * for encoding
     */
    public AudioEncoderDecoder(){
        audioAttr.setBitRate(bitrate);
        audioAttr.setChannels(channels);
        audioAttr.setSamplingRate(samplingRate);
    }

    public void encodeAudio(File source, File target, String mimeType){
        //Change the hardcoded mime type later on
        if(mimeType.equals("audio/mp3")){
            this.mp3ToOgg(source, target);
        }
    }

    private void mp3ToOgg(File source, File target){
        //ADD CODE FOR CHANGING THE EXTENSION OF THE FILE
        encoAttrs.setFormat(oggFormat);
        audioAttr.setCodec(oggCodec);
        encoAttrs.setAudioAttributes(audioAttr);
        try{
        encoder.encode(source, target, encoAttrs);
        }catch(Exception e){
            System.out.println("Encoding Failed");
        }
    }

    private void oggToMp3(){
        //ADD CODE FOR CHANGING THE EXTENSION OF THE FILE
        encoAttrs.setFormat(mp3Format);
        audioAttr.setCodec(mp3Codec);
        encoAttrs.setAudioAttributes(audioAttr);
        try{
        encoder.encode(source, target, encoAttrs);
        }catch(Exception e){
            System.out.println("Encoding Failed");
        }
    }

    public static void main(String[] args){
        AudioEncoderDecoder aed = new AudioEncoderDecoder();
        File source = new  File("singelementsmp3.mp3");
        File target = new File("test.ogg");

        //Test Mp3 To Ogg Convertion
        String mimeType = "audio/mp3";
        aed.encodeAudio(source, target, mimeType);

        //Test Ogg To Mp3 Convertion
    }
}

Now, the xToX conversion basically the same throughout the whole code implementation. The only thing will change is the codec.

What is another way to make this code cleaner? From my point of view it repeats itself. What should I change? Or are there any design patterns I should implement?

Overall

Design first.

Name and semantic of your class

  I have this code that converts audio to different file formats.

Not exactly, your code is a client to an Encoder, that converts audio to different file formats.
In design, it is very important to be certain about the reason and semantic of an object. So first, the name AudioEncoderDecoder is bad. It says nothing about the sematic of the object. I would call it AudioFormatConverter as it converts one audio file format to another.

Public methods

public void encodeAudio(File source, File target, String mimeType);

a) Naming encodeAudio() is not clear, convert() would be clear and consistent with the classname.

b) Parameters: source, target file seems good. But why would I pass in the mimeType? And how can I specify which target format I choose?

Expected method signature:

public void convert(File source, File target) throws AudioConversionException();

One conversion method in the public interface, source and target file, the mime type is guessed by the file extension. You could add another public method, where you take in the mimetype, as needed. But keep the interface of your class as simple, logic, consistent and understandable as possible.

I would declare an checked exception for error handling, formats not avail or whatever.

Design Patterns

If there will be more formats, codecs, mimetypes you could take a look at Chain of responsibility or Dispatcher/Command pattern.
These patterns also add flexibility to easily support more mimetypes at runtime.
Without those patterns, you may get a mid-to-big if/then/else block.

Possible Design with Command, Strategy pattern applied:

AudioFormatConverters interface is format-agnostic, this means it is possible to implement a converter, that doesn't use Files but only InputStreams or byte[] arrays.
Concrete AudioFileFormatConverter converts from and to Files, and holds a list of supported ConversionCommands
ConversionCommand encapsulates logic to convert things, a client can check via isSupported(source, target) if the source-format and target-format is supported by this command (handler would by a common name, too)
Mp3ToOggConversionCommand Every conversion is a command, one can decide that this is too much, but with that you can save state in the command, like executation duration, error states, sizes, redo, undo, execute asynchronous...
ConversionStrategy with Impl EncoderConversionStrategy encapsulates the concrete implementation of conversion, that is your duplicate code, this strategy how to convert 2 files by using the Encoder

Note that ConversionCommands should be prototypes, so that each execute happens in its own Command instance.

Now in code, you would setup the AudioFileFormatConverter and add the supported ConversionCommands to it.

Existing code

Obey the basic OO principles.

DRY - Don't repeat yourself. If you find code is duplicate, introduce
abstractions or extract methods.
Use interfaces

In your case you can easily improve the code if you generalize the mp3ToOgg and oggToMp3 methods.

So that you have only one general method for converting with the signature:

private void convert(File source, File target, String format, String codec);


Sort all even numbers in ascending order and then sort all odd numbers in Descending order Java Given an array of integers (both odd and even), the task is to sort
  them in such a way that the first part of the array contains odd
  numbers sorted in descending order, rest portion contains even numbers
  sorted in ascending order.

  Examples:

Input  :

A[] = {1, 2, 3, 5, 4, 7, 10}

Output :

A[] = {7, 5, 3, 1, 2, 4, 10}

Input  :

A[] = {0, 4, 5, 3, 7, 2, 1}

Output :

A[] = {7, 5, 3, 1, 0, 2, 4}

  Input:

  The first line of input contains an integer T denoting the no
  of test cases. Then T test cases follow. Each test case contains an
  integer N denoting the size of the array. The next line contains N
  space separated values of the array.

  Output:

  For each test case in a new line print the space separated
  values of the  new transformed array.

Constraints:

1<=T<=100

1<=N<=100 0

0<=A[]<=100

  Example:

  Input:

  2
  7
  1 2 3 5 4 7 10
  7
  0 4 5 3 7 2 1

  Output:

  7 5 3 1 2 4

  10 7 5 3 1 0 2 4

My approach:

import java.util.Scanner;
import java.util.List;
import java.util.ArrayList;
import java.util.Collections;

class GFG {

    private static List<Integer> sortListOrder (List<Integer> list) {
        List<Integer> oddList = new ArrayList<>();
        List<Integer> evenList = new ArrayList<>();

        for (Integer elem : list) {
            if (elem%2 != 0) {
                oddList.add(elem);
            }
            else {
                evenList.add(elem);
            }
        }
        Collections.sort(evenList);
        Collections.sort(oddList);
        List<Integer> reversedList = new ArrayList<>();
        for (int i = oddList.size() - 1; i >= 0; i--) {
            reversedList.add(oddList.get(i));
        }

        reversedList.addAll(evenList);

        return reversedList;
    }

    public static void main (String[] args) {
        Scanner sc = new Scanner(System.in);
        int numTests = sc.nextInt();

        while(numTests-- > 0) {
            int size = sc.nextInt();
            List<Integer> list = new ArrayList<>();

            for (int i = 0; i < size; i++) {
                int elem = sc.nextInt();
                list.add(elem);
            }

            List<Integer> sortedList = sortListOrder(list);

            for (Integer elem : sortedList) {
                System.out.print(elem + " ");
            }
            System.out.println();

        }
        sc.close();
    }
}

I have the following questions with regards to the above code:

How can I further improve my approach?
Is there a better way to solve this question?
Are there any grave code violations that I have committed?
Can space and time complexity be further improved?

Reference

You already dug into the standard library a bit (which is good) and discovered the beautiful sort method. Using this method is a perfect start.

What you overlooked is, that there is an overloaded second version of sort which accepts a comparator, so that you can perform all kinds of custom sorting with a single call.

Thus, instead of splitting the list in even and odd parts, sorting them separately, reversing one part, gluing it all back together, you could simply have created a comparator which encapsulates the logic, passed this into the sort method, and be done.

(BTW: even if you take your approach, reversing a list should be done with Collections.reverse())

So: create a comparator, do it all in a single sort call, and I look forward to see your revamped solution here in a day or so. And while you are at it: "GFG" is not an acceptable class name: what does that class do? Give it a name worth remembering!


Pairwise Combinations Generator Below is an algorithm to generate all pairwise combinations for a set of random variables. See here for an explanation of what constitutes a minimal all-pairs set. The algorithm below works but I feel that it is inefficient. Any suggestions would be welcome.

from operator import itemgetter
from itertools import combinations, product, zip_longest, chain, permutations

def pairwise(*variables, **kwargs):
    num_vars = len(variables)

    #Organize the variable states into pairs for comparisons
    indexed_vars = sorted(
        ([(index, v) for v in vals] for index, vals in enumerate(variables)),
        key=len,
        reverse=True)

    var_combos = combinations(range(num_vars), 2)
    var_products = (product(indexed_vars[i], indexed_vars[j]) for i, j in var_combos)
    vars = chain.from_iterable(zip_longest(*var_products))

    #Initialize the builder list
    builders = []
    seen_pairs = set()

    #Main algorithm loop
    for pair in vars:
        if not pair or pair in seen_pairs:
            continue

        d_pair = dict(pair)

        #Linear search through builders to find an empty slot
        #First pass: look for sets that have intersections
        for builder in builders:
            intersection = builder.keys() & d_pair.keys()
            if len(intersection) == 1:
                v = intersection.pop()
                if builder[v] == d_pair[v]:
                    builder.update(d_pair)
                    #Update seen pairs
                    seen_pairs.update(combinations(builder.items(), 2))
                    break
        else:
            #Second Pass
            for builder in builders:
                intersection = builder.keys() & d_pair.keys()
                if not len(intersection):
                    builder.update(d_pair)
                    #Update seen pairs
                    seen_pairs.update(combinations(builder.items(), 2))
                    break
            else:
                #No empty slots/pairs identified. Make a new builder
                builders.append(d_pair)
                seen_pairs.add(pair)

    #Fill in remaining values
    complete_set = set(range(num_vars))
    defaults = {var[0][0]: var[0][1] for var in indexed_vars}

    for builder in builders:
        if len(builder) == num_vars:
            yield tuple(val for index, val in sorted(builder.items()))
        else:
            for key in complete_set - builder.keys():
                builder[key] = defaults[key]
            yield tuple(val for index, val in sorted(builder.items()))

Usage Example:

u_vars = [
    [1, 2, 3, 4],
    ["A", "B", "C"],
    ["a", "b"],
    ]

result = list(pairwise(*u_vars))
print(result) #prints list of length 12

This is not generating correct pairwise combinations.

Input data

[['Requests'],
    ['Create','Edit'],
    ['Contact Name','Email','Phone','Subject','Description','Status','Product Name','Request Owner','Created By','Modified By','Request Id','Resolution','To Address','Account Name','Priority','Channel','Category','Sub Category'],
    ['None','is','isn\'t','contains','doesn\'t contain','starts with','ends with','is empty','is not empty'],
    ['Contact Name','Email','Phone','Subject','Description','Status','Product Name','Request Owner','Mark Spam','Resolution','To Address','Due Date','Priority','Channel','Category','Sub Category']]

Result by Microsoft PICT

http://pastebin.com/cZdND9UA

Result by your code
http://pastebin.com/EC6xv4vG

This generates only for NONE in most of cases such as this

Requests    Create  Account Name    None    Category
Requests    Create  Account Name    None    Channel
Requests    Create  Account Name    None    Description
Requests    Create  Account Name    None    Due Date


Could my object use static methods? Anything I need to do to make the code better? With the Zend coding convention in the back of my mind, I have set this up.

It lets you communicate with the API of the vendor TargetSMS. They allow you to bill customers trough SMS. For now I've only spent time on the NonSubscription option.

I would just like to know if it's ok, or what could be better.  I am mostly interested in knowing if the TargetSms object could use static methods.  For example, I think the isAllowedIp method could be static, since I would like to use it even if the object is not initiated (I was told that's the idea behind static methods).

TargetSMS object

<?php
/**
* TargetSMS object with TargetSMS related methods.
*/

namespace TargetPay\Sms;

class TargetSms
{
    /**
* The allowed IP of TargetSms.
* @var array
*/
    protected $_targetSmsIp = array('89.184.168.65');

    /**
* The ok response code for TargetSMS.
* @var number
*/
    protected $_responseCode = 45000;

    /**
* Check if the request is comming from TargetSMS.
* @param string $ip
* @return boolean
*/
    public function isAllowedIp($ip = '')
    {
        if (in_array($ip, $this->_targetSmsIp)) {
            return true;
        }
        return false;
    }

    /**
* Get the TargetSMS required responsecode.
* @return number
*/
    public function getResponseCode()
    {
        return $this->_responseCode;
    }

    /**
* Add a new allowed ip address to the array with allowed ip addresses.
* @param string $ip
*/
    public function addAllowedIp($ip = '')
    {
        $this->_targetSmsIp[] = $ip;
    }
}

If you wanted to make isAllowdIp static (and there's nothing wrong with that, per se), you -indeed- don't need an instance of your class to call that method. But owing to there being no instance, you won't have access to any non-static properties of your class, either.
To get around that, you'd have to change:

protected $_targetSmsIp = array('89.184.168.65');

To

protected static $_targetSmsIp = array('89.184.168.65');

And change these method, too:

public static function isAllowedIp($ip)
{//don't check empty strings, they're always invalid ips!
    return !!in_array($ip, self::$_targetSmsIp);
}
public static function addAllowedIp($ip)
{//don't allow '' defaults! adding empty strings aren't valid ips
    if (!filter_var($ip, FILTER_VALIDATE_IP))
    {//check what you're adding to the OK-list!
        throw new InvalidArgumentException($ip. ' is not a valid IP');
    }
    self::$_targetSmsIp[] = $ip;
}

This, by itself isn't code that makes my eyes water, but it doesn't exactly sit well with me, either: It's hard to tell what the actual task of your class is: validate IP's/data? Is it the API connection layer?
I gather it's the latter. In which case, I'd define my methods to only accept objects of a given type. This object is where you can filter, check and validate all input... For example, the IpObject:

class IpObject
{
    private $ip = null;
    private static $valideIps = array();
    public function __construct($ip = null)
    {
        if ($ip)
        {
            $this->setIp($ip);
        }
    }
    public function getIp()
    {
        return $this->ip;
    }
    public function setIp($ip)
    {
        if (!filter_var($ip, FILTER_VALIDATE_IP) || !in_array($ip, self::$validIps))
        {
            throw new InvalidArgumentException($ip.' is not valid');
        }
        $this->ip = $ip;
        return $this;
    }
}

The major problem with your creating statics here is that, by changeing the allowed IP's for one instance, you're changing the allowed IP's accross the board: you can't black-list or OK ip's for instances of your object individually, so after a while it'll get quite tricky to work out which IP's are allowed and which aren't.

All in all, an array in this object won't add too much overhead... not that you notice, and it will make life easier when testing/debugging.
Instead of using statics. having an instance at the ready is 99.99% of times the better option.


Bruteforce XORing speed I'm trying to learn more efficient coding so I'm doing some challenges. I have a challenge where I'm given some hex data. I know that hex data is XOR encoded with a four character long ASCII key. The only thing I know about the clear text data is that it contains "a lot of '#' and space characters".

I don't know what they mean by a lot and there are many possibilities that end up with more than 50 hash tags and spaces, though most of them have less than 10 of each. So I figured the only way I could do this would be to narrow down the number of possibilities.

In the following code I'm iterating through every possible combination of 4 character ASCII using the Cartesian product (keys). I'm using a namedtuple to store the number of hashtags and spaces for each key that has more than 10 of either. I'm hoping this would help me determine good potential candidates to manually sift through.

import itertools,re,string
from collections import namedtuple

# key is four ascii chars long so we will iterate through data 4-char chunk at a time
data = re.findall('....?', '\x6e\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x47\x45\x50\x02\x44\x46\x53\x01\x44\x45\x50\x02\x47\x45\x50\x01\x44\x46\x53\x02\x47\x45\x50\x02\x6e\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x47\x45\x53\x01\x47\x45\x53\x01\x44\x45\x50\x01\x44\x46\x53\x01\x44\x46\x50\x02\x44\x46\x53\x01\x6e\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x47\x45\x53\x01\x44\x45\x53\x01\x44\x45\x50\x02\x47\x45\x50\x01\x44\x46\x53\x02\x47\x45\x50\x01\x6e\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x47\x45\x53\x01\x47\x45\x53\x01\x44\x45\x50\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x50\x02\x6e\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x47\x45\x50\x02\x44\x46\x53\x01\x44\x45\x50\x01\x44\x46\x53\x01\x44\x46\x50\x02\x47\x45\x50\x01\x6e\x6c\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x6c\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x45\x50\x01\x44\x46\x30\x4e\x09\x0b\x06\x4f\x0d\x05\x12\x55\x0d\x09\x1d\x52\x44\x25\x1c\x4f\x10\x14\x1c\x4d\x44\x46\x53\x02\x47\x6c\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x46\x53\x01\x44\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x45\x50\x02\x47\x6c\x79\x6c\x01\x08\x06\x2b\x49\x4b\x5e\x0c\x6e\x6c\x42\x0f\x44\x35\x07\x40\x16\x12\x06\x51\x44\x05\x1c\x4c\x09\x13\x1d\x48\x07\x07\x07\x48\x0b\x08\x00\x0f\x6e\x54\x5d\x01\x37\x0e\x06\x55\x00\x09\x04\x4f\x44\x05\x1c\x4c\x09\x13\x1d\x48\x07\x07\x07\x48\x0b\x08\x00\x0f\x6e\x55\x5d\x01\x28\x09\x14\x4e\x11\x12\x5d\x2b\x6e\x36\x1f\x44\x05\x15\x16\x01\x17\x03\x1f\x44\x07\x12\x53\x58\x0b\x13\x01\x01\x05\x05\x07\x48\x0b\x08\x53\x1f\x6e')

# Cartesian product, all possible printable ascii chars
keys = itertools.product(string.printable, repeat = 4)

# Namedtuple and list used to store good candidates
Possible = namedtuple('Possible', ['key', 'hash', 'space'])
possibles = []

for k in keys:
    clear = ''
    key = ''.join(x for x in k) # product() returns tuples, need to join them
    for word in data:
        clear +=  bytearray(a^b for a, b in zip(*map(bytearray, [word, key])))

    # Count hash tags and spaces and store good candidates
    hcount = clear.count('#')
    scount = clear.count(' ')
    if hcount > 10 or scount > 10:
        possibles.append( Possible(key, hcount, scount) )

# display candidates
for possible in possibles:
    print possible

My code seems to have a very poor performance and I would love to know how I can improve it.

The very first thing I do when I'm running into performance issues with my code is answer three questions:

How fast does my code need to be?
How fast is it now?
What is slowing me down?

You're the only one who can provide an answer to the first one, so I'll pass on that for now.  The second and third ones can be answered simultaneously while profiling your code, using cProfile.  I saved your code as "bruteforce_xor.py", so by running

C:\Users\Dannnno\Desktop\> python -m cProfile bruteforce_xor.py

I was able to find out how long it takes to run in total, and what is actually taking so long.  This was the output:

  I don't have any! I let this run for a while and it never finished, which shouldn't come as a surprise once you finish reading below.

Before we apply the (lack of) results of our profiling, lets look at some general improvements.

Imports

In general, each import deserves its own line.  This helps with readability

import itertools
import re
import string
from collections import namedtuple

Whitespace

You generally want whitespace around operators, commas, etc.  Exceptions include the = sign for default values and keyword arguments, and for precedence.

On the flip side, don't put them on the outside of method calls, like

possibles.append( Possible(key, hcount, scount) )

instead, use

possibles.append(Possible(key, hcount, scount))

Naming

Some of your variables could be more clearly named.   Instead of data, I'd call that chunks.  Instead of Possible I'd use Candidate, and possibles -> candidates.  Instead of for k in keys just type for key in keys.  Things of that nature.

Regex

I question the regex you're using...  The regex '....?' will match any 3 or 4 character chunk, because you've made the last one optional.  Based on your comments, you only want 4 character chunks.

Itertools

You use itertools some of the time, but not always.  Use it for zipping and mapping as well.  That being said, I don't really see the point of

map(bytearray, [word, key])

Usually I prefer reserving higher order functions for cases where I don't actually know how many elements there are.  In this case

bytearray(word), bytearray(key)

seems much cleaner

Your speed is going to be very, very hard to improve if you keep using a brute-force algorithm.

string.printable consists of 100 ASCII characters.  When you get the cartesian product of that with itself (4 times) you get 100,000,000 (!!!) combinations.  Then you iterate over each of those combinations.

Inside of that iteration, you then iterate over every "word" in your data.  Using re.findall('....?', blah) you're going to generate 1889 chunks of length 4, and 1890 chunks of length 3, for a grand total of 3779 chunks.  This means that you're essentially doing 377,900,000,000 (!!!!!) iterations of your loop.

That's bad.  That's really bad.  The number one thing you absolutely must try to improve is that - find another method that reduces how many times you look at everything.  I don't have a good suggestion for that - I don't know enough about your problem or what (if any) expectations you can have for the input to provide a good way to filter that.  However, I can give you some other suggestions.

Use lazy evaluation

Like I mentioned earlier, don't call zip and map.  Instead defer the calculations to when you actually need them using itertools.izip and itertools.imap.

Don't repeat calculations

You transform every chunk you have into a bytearray every single time you go through your loop.  Instead of repeating that calculation, why don't you convert them first?  Same for your cartesian products (except here I'd probably convert your string.printable and not the cartesian products themselves)

Finally

Bitwise operations are generally executed in constant time so there isn't a whole lot you can do about that.  All I might suggest is using an LRU cache, but honestly I don't see that speeding up the process.

As a final piece of food for thought, this (poorly timed) code runs (on my admittedly slow netbook) as follows:

import time

start = time.time()
# 377,900,000,000 is too big for an arg to xrange in Python 2.7k
for j in xrange(1000):
    for i in xrange(377900000):
        j ^ i
diff = time.time() - start
print diff

Output:

  None, again.  I ran this for 5 minutes on (again, admittedly crummy netbook) and it couldn't finish.  This is the equivalent number of iterations with a single constant time operation.


Optimizing python code for big data sets I'm trying to optimize a python string that works on big data sets, the way it works is by taking in a with a list of keywords and scores and taking in a file loaded with data from the twitter api. The program does a keyword match against tweet text. At the end of the program I want to produce an average for each term found in text object of the json file. e.g.

sad 3

With sad being the keyword and 3 being the average score.

It's running way too slow but I'm new to Python coming from a php background and I think I'm doing things the php way in python.

How can I get this code to run faster?

import sys
import json
import re

def findRecord(key, records):
    for r in records:
        if r[0] == key:
            return r

def average_records(records):
    for r in records:
        if r[1] > 0:
            avg = r[1] / r[2]
            print r[0] + ' ' + str(avg)
        else:
            avg = r[3] / r[4]
            print r[0] + ' ' + str(avg)

def hw(sent_file, tweet_file):
    scores = {}

    sent_file = open(sent_file, 'r')

    for line in sent_file:
        term, score = line.split("\t")
        scores[term] = int(score)

    recored_affin = []

    #print scores.items()

    data = []

    with open(tweet_file, 'r') as f:
        for line in f:
            data.append(json.loads(line))

        #print data[4]['text']

        for tweet in data:
            total = 0
            if 'text' in tweet:
                for k, v in scores.iteritems():

                    #print tweet['text']
                    num_of_aff = len(re.findall(k, tweet['text']))
                    if num_of_aff > 0:
                        #print "Number is: " + str(num_of_aff)
                        #print "Word is: " + k
                        #print "Tweet is: " + tweet['text']
                        total += (v * num_of_aff)
                        #print "Score is: " + str(total)

                        #while count < len(recorded_affin):

                        foundRow = findRecord(k, recored_affin)

                        if foundRow != None:
                            index = recored_affin.index(foundRow)
                            quick_rec = recored_affin[index]

                            if v > 0:
                                new_value = quick_rec[1] + v
                                new_count = quick_rec[2] + 1
                                old_neg_value = 0
                                old_neg_count = 0
                                recored_affin.append([k, new_value, new_count, old_neg_value, old_neg_count])
                                recored_affin.remove(foundRow)
                            elif v < 0:

                                old_pos_value = 0
                                old_pos_count = 0
                                new_value = quick_rec[3] + v
                                new_count = quick_rec[4] + 1
                                recored_affin.append([k, old_pos_value, old_pos_count, new_value, new_count])
                                recored_affin.remove(foundRow)

                        else:
                            if v > 0:
                                recored_affin.append([k,v,1,0,0])
                            elif v < 0:
                                recored_affin.append([k,0,0,v,1])

                            #print recored_affin

                        ##print foundRow

                ##print total
    average_records(recored_affin)

def lines(fp):
    print str(len(fp.readlines()))

def main():
    sent_file = open(sys.argv[1])
    tweet_file = open(sys.argv[2])
    hw(sys.argv[1], sys.argv[2])
    #lines(sent_file)
    #lines(tweet_file)

if __name__ == '__main__':
    main()

Your code contains a mix of tabs and spaces. This caused your code to display incorrectly before I edited it. The most common way in python is to use only spaces. You should be able to configure your editor to insert spaces instead of tabs when you push the tab key.

import sys
import json
import re

def findRecord(key, records):

Python convention is to name function lowercase_with_underscores

    for r in records:
        if r[0] == key:
            return r

It is going to be inefficient to loop over records looking for things like this. Instead, you use a dictionary and look them up by key.

def average_records(records):
    for r in records:

Rather than indexing r all over the place, I suggest using:

    for k, new_value, new_count, old_neg_value, old_neg_count in records:

Then you can access those names directly. It'll be easier to read and probably marginally faster.

        if r[1] > 0:
            avg = r[1] / r[2]
            print r[0] + ' ' + str(avg)
        else:
            avg = r[3] / r[4]
            print r[0] + ' ' + str(avg)

In this case, you can do: print r[0], avg for the same result.

def hw(sent_file, tweet_file):

I have no idea what hw means

    scores = {}

    sent_file = open(sent_file, 'r')

    for line in sent_file:
        term, score = line.split("\t")
        scores[term] = int(score)

This scores bit is a nicely self contained section. I suggest making it a separate function.

    recored_affin = []

    #print scores.items()

Don't keep dead code, just remove it. If you think you might need it back look into version control.

    data = []

    with open(tweet_file, 'r') as f:
        for line in f:
            data.append(json.loads(line))

        #print data[4]['text']

You're finished with the file now, you should really drop out of the with block.

        for tweet in data:

There isn't really much point in storing the json objects in a list just to process them. Just process them as you get them.

            total = 0
            if 'text' in tweet:
                for k, v in scores.iteritems():

                    #print tweet['text']
                    num_of_aff = len(re.findall(k, tweet['text']))
                    if num_of_aff > 0:
                        #print "Number is: " + str(num_of_aff)
                        #print "Word is: " + k
                        #print "Tweet is: " + tweet['text']
                        total += (v * num_of_aff)
                        #print "Score is: " + str(total)

                        #while count < len(recorded_affin):

Don't leave commented code in there

                        foundRow = findRecord(k, recored_affin)

                        if foundRow != None:

Use is None to check for foundRow

                            index = recored_affin.index(foundRow)

That's going to be expensive, it scans through the whole list again.

                            quick_rec = recored_affin[index]

Isn't this just foundRow again?

                            if v > 0:
                                new_value = quick_rec[1] + v
                                new_count = quick_rec[2] + 1
                                old_neg_value = 0
                                old_neg_count = 0
                                recored_affin.append([k, new_value, new_count, old_neg_value, old_neg_count])
                                recored_affin.remove(foundRow)

Expensive, has to scan through again.
                                elif v < 0:

                                old_pos_value = 0
                                old_pos_count = 0
                                new_value = quick_rec[3] + v
                                new_count = quick_rec[4] + 1
                                recored_affin.append([k, old_pos_value, old_pos_count, new_value, new_count])
                                recored_affin.remove(foundRow)

You've got some duplication here, you should move the common logic out of the if blocks.

                        else:
                            if v > 0:
                                recored_affin.append([k,v,1,0,0])
                            elif v < 0:
                                recored_affin.append([k,0,0,v,1])

                            #print recored_affin

                        ##print foundRow

                ##print total
    average_records(recored_affin)

def lines(fp):
    print str(len(fp.readlines()))

def main():
    sent_file = open(sys.argv[1])
    tweet_file = open(sys.argv[2])

Why do you open these file but never do anything with them?

    hw(sys.argv[1], sys.argv[2])
    #lines(sent_file)
    #lines(tweet_file)

if __name__ == '__main__':
    main()

Your speed issues are probably the result of using a list and constantly searching over the whole list instead of using a dictionary. Make recorded_affin a dictionary, and your code should be simpler and faster.


Updating or adding data to an associative array as a cache I have this if else statement where I don't know how to remove the code duplication

if(array_key_exists($key,$cachetable))
{
 if($cachetable[$key] != $newdata[$key])
  {
    //data existed but is not valid
    //update the cachetable
    $cachetable[$key] == $newdata[$key]
  }
}

else
{
    //data didn't existed before in cache so add it now
    //update the cachetable
    $cachetable[$key] == $newdata[$key]
 }

Modify the code to this:

//just use the AND operator (&&) to check if it's both existing AND valid
//use strict comparison (===) whenever possible
if(array_key_exists($key,$cachetable) && $cachetable[$key] === $newdata[$key]){
    //data exists and valid
} else {
    //data doesn't exist or isn't valid
    //to assign data, use a single equal (=).
    //A double equal (==) is a loose comparison
    $cachetable[$key] = $newdata[$key];
 }

//this one checks if it's not in the array
//or if it does, it checks if it's not valid
//when either one is true, the condition passes
if(!array_key_exists($key,$cachetable) || $cachetable[$key] !== $newdata[$key]){
    //data doesn't exists or isn't valid
}


Fast Risk Battle Simulator I have written some code in python to determine the outcome(including the remaining troops of both sides) of a Risk battle fought until one side is defeated. When using the Numba's njit decorator the code can simulate one battle's outcome in ~ 0.2ms.
This is not fast enough for the application I have in mind. Ideally it would be one to two orders of magnitude faster. Any advice on how to improve performance would be greatly appreciated.
Function Code:
@njit
def _attack_(n_atk,n_def):
    n_atk_dies = 0
    n_def_dies = 0
    while n_atk >0 and n_def>0:
        if n_atk>=3:
            n_atk_dies = 3
        if n_atk==2:
            n_atk_dies = 2
        if n_atk ==1:
            n_atk_dies = 1
        if n_def >=2:
            n_def_dies = 2
        if n_def ==1:
            n_def_dies = 1
        atk_rolls = -np.sort(-np.random.randint(1,7,n_atk_dies))
        def_rolls = -np.sort(-np.random.randint(1,7,n_def_dies))
        comp_ind = min(n_def_dies,n_atk_dies)
        comp = atk_rolls[:comp_ind]-def_rolls[:comp_ind]
        atk_losses =len([1 for i in comp if i<=0])
        def_losses = len([1 for i in comp if i>0])
        n_atk -= atk_losses
        n_def -= def_losses
    return [n_atk,n_def]

Benchmarking Code:
atk_list = list(np.random.randint(1,200,10000))
def_list = list(np.random.randint(1,200,10000))
cProfile.run('list(map(_attack_,atk_list,def_list))',sort='tottime')

You play with up to 199 dice per player. So most of the time, the attacker will attack with 3 dice and the defender will attack with 2 dice. As long as that's the case, you could simply pick one of the three possible outcomes (with precalculated probabilities) and apply it. Outline:
def _attack_(n_atk,n_def):
    while n_atk >= 3 and n_def >= 2:
        # pick random outcome and update the dice numbers
    while n_atk >0 and n_def>0:
        # same as before
    return [n_atk,n_def]

You could even precompute the result distributions for all \$199^2\$ possible inputs and randomly pick the result according to the given input's distribution. Outline:
def _attack_(n_atk,n_def):
    return random.choices(results[n_atk, n_def], cum_weights=cum_weights[n_atk, n_def])