A C++ function to read Code Points from an UTF-8 Stream I've written a function that reads and returns one UTF-8 code point from an istream. I am wondering if the code is efficient or if there are some obvious problems with the implementation.
chr_t utf32::get_utf32_char(std::istream &in_stream) {
    int next;
    chr_t out = in_stream.get();
    if (out == -1 || out < 0x80) {
        return out;
    } else if ((out & 0xe0) == 0xc0) {
        out &= 0x1f;
        out <<= 6;
        next = in_stream.get();
        if (next == -1) goto invalid_seq;
        out |= next & 0x3F;
        return out;
    } else if ((out & 0xf0) == 0xe0) {
        out &= 0x0f;
        out <<= 12;
        next = in_stream.get();
        if (next == -1) goto invalid_seq;
        out |= (next & 0x3F) << 6;
        next = in_stream.get();
        if (next == -1) goto invalid_seq;
        out |= next & 0x3F;
        return out;
    } else if ((out & 0xf8) == 0xf0) {
        out &= 0x07;
        out <<= 18;
        next = in_stream.get();
        if (next == -1) goto invalid_seq;
        out |= (next & 0x3F) << 12;
        next = in_stream.get();
        if (next == -1) goto invalid_seq;
        out |= (next & 0x3F) << 6;
        next = in_stream.get();
        if (next == -1) goto invalid_seq;
        out |= next & 0x3F;
        return out;
    } else {
        throw std::runtime_error("invalid utf8 character");
    }
invalid_seq:
    throw std::runtime_error("unexpected end of utf8 sequence");
}

Overview
There is a lot of repeated code that could be removed by use of functions.
When bittwiddling like this it would be nice for a human readable explanation of what you are doing. I had to look up the unicode spec to make sure you were doing it correctly.
A lot of UTF-8 files (stream) contain a BOM marker 0xEF, 0xBB, 0xBF as the first code point. This is not part of the text stream and should be discarded if it exists. Though you may do this at the layer of abstraction above this in which case a comment pointing out that the BOM marker is not removed should be added.
You don't validate that the bytes 2 through 4 have the correct pattern for UTF-8 you just make that assumption.
You use exceptions on streams. Normally you would mark the stream as bad and return. The user of the stream is supposed to check the state of the stream before using any output (and further reading will fail).
C++ uses operator>> to read from a stream. It would be nice to be able to read your characters using this operator.
Code Review
The name of the function is not quite correct:
chr_t utf32::get_utf32_char(std::istream &in_stream)

Code points are distinct from there encoding. You are converting an code point that was encoding UTF-8 into UCS-4 (not UTF-32). UTF-32 is another encoding format used for transportation. I would note that UCS-4 and UTF-32 look the same but they are not the same thing.

You read into next (an int) in all locations apart from here:
    int next;
    chr_t out = in_stream.get();

Why not be consistent. I especially worry about corner case and auto conversions with characters and integers. Can't think of anything that would go wrong but why risk it. Read into next (the int) check for EOF then convert to your character representation.

Don't use magic numbers. In this context you should use EOF (not -1).
    if (out == -1 || out < 0x80) {
        return out;

I hate else on the same line as }.
    } else if ((out & 0xe0) == 0xc0) {

But your code your style.
Very few coding standards use this system.
In my opinion (so ignorable) you don't need to crush the code together that much. Extra vertical spacing will make the code easier to read.

Questionablt use of goto:
        if (next == -1) goto invalid_seq;

Why not simply:
        if (next == EOF) {
            throw std::runtime_error(unexpectedESFMessage);
        }

Redesign:
I would have used a more data driven approach:
struct Encoding
{
    char    mask;
    char    value;
    int     extra;
};
Encoding const utf8Info[] = {
                        {0x80, 0x00, 0}
                        {0xE0, 0xC0, 1}
                        {0xF0, 0xE0, 2}
                        {0xF8, 0xF0, 3}
                      };
chr_t decodeUtf(std::istream& stream, chr_t result, int count)
{
    for(; count; --count) {
        int next = stream.get();
        if (next & 0xC0 != 0x80) {
            // Not a valid continuation character
            stream.setstate(std::ios::badbit)
            return -1;
        }
        result = (result << 6) | (next & 0x3F);
    }
    return result;
}
chr_t getCodePoint(std::istream& stream)
{
    // NOTE: Does not remove any initial BOM marker.

    int next = stream.get();
    if (next == EOF) {
        return -1;
    }
    for(auto const& type: utf8Info) {
        if ( next & type.mask == type.value ) {
           return decodeUtf(stream, next & ~type.mask, type.extra);
        }
    }
    // Not a valid first character
    stream.setstate(std::ios::badbit)
    return -1;
}

std::istream& operator>>(std::istream& str, chr_t& out)
{
    chr_t tmp = getCodePoint(str);
    if (str) {
       out = tmp;
    }
    return str;
}

.


Check before proceeding if the task in BackgroundWorker has completed I have a Login Form (which has a label that will show status of background process, username and password text boxes, Login and Cancel buttons) along with that I have a background worker to do some time taking task (in the meanwhile when Login Form loads and user provide details in text boxes).

Background task keep running while the user will be providing username and password on Login Form. As soon as the user hits the Login button I want to check the status if the background process has completed successfully and then only it should proceed (launch the MainForm)

Otherwise should pop-up error. Following is what I did to achieve but the 'while()' strategy doesn't look elegant. Is there any more appropriate way?

  public partial class LoginForm : Form
    {
        private static BackgroundWorker bw = new BackgroundWorker();
        private static ManualResetEvent mre = new ManualResetEvent(false);

        enum status
        {
            Blank,
            Success,
            Error
        };

        public LoginForm()
        {
            InterimProceedings();
            InitializeComponent();
        }

        private void InterimProceedings()
        {

            bw.DoWork += new DoWorkEventHandler(bw_DoWork);
            bw.RunWorkerCompleted +=
            new RunWorkerCompletedEventHandler(bw_RunWorkerCompleted);

            if (!bw.IsBusy)
            {
                bw.RunWorkerAsync();
            }
            else
            {
                throw new InvalidOperationException("BackgroundWorker is busy");
            }
        }

        private static void bw_DoWork(object sender, DoWorkEventArgs e)
        {
          //  Time taking task
              mre.Set();
        }

        private void bw_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
        {
            if (!(e.Error == null))
            {
                this.lbl_status.Text = "Cannot proceed, Error occured";
                appStatus = status.Error;
            }
            else
            {
                this.lbl_status.Text = "Good to go...";
                appStatus = status.Success;
            }
        }

        private void btn_login_Click(object sender, EventArgs e)
        {
            mre.WaitOne();

            if(appStatus == status.Success)
            {
                // Proceed with intended work
            }

            else
            {
                // Pop-up error occurred
            }
        }
}

You can use ManualResetEvent to wait until background work is done. So, you can define in LoginForm as:

private static ManualResetEvent mre = new ManualResetEvent(false);

When background work is completed, you should change the state of mre from the thread method bw_DoWork and update appStatus:

mre.Set();

btn_login_Click should wait for mre instead of while loop:

mre.WaitOne();

The complete code:

public partial class LoginForm : Form
{
    private static BackgroundWorker bw = new BackgroundWorker();
    private static ManualResetEvent mre = new ManualResetEvent(false);

    enum status
    {
        Blank,
        Success,
        Error
    };

    public LoginForm()
    {
        InterimProceedings();
        InitializeComponent();
    }

    private void InterimProceedings()
    {
        bw.DoWork += new DoWorkEventHandler(bw_DoWork);
        bw.RunWorkerCompleted += new RunWorkerCompletedEventHandler(bw_RunWorkerCompleted);

        if (!bw.IsBusy)
        {
            bw.RunWorkerAsync();
        }
        else
        {
            throw new InvalidOperationException("BackgroundWorker is busy");
        }
    }

    private void bw_DoWork(object sender, DoWorkEventArgs e)
    {
      //  Time taking task
      //  Update appStatus
      appStatus = status.Success;
      mre.Set();
    }

    private void bw_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
    {
        if (!(e.Error == null))
        {
            this.lbl_status.Text = "Cannot proceed, Error occured";
            appStatus = status.Error;
        }
        else
        {
            this.lbl_status.Text = "Good to go...";
            appStatus = status.Success;
        }
    }

    private void btn_login_Click(object sender, EventArgs e)
    {
        mre.WaitOne();

        if(appStatus == status.Success)
        {
            // Proceed with intended work
        }

        else
        {
            // Pop-up error occurred
        }
    }
}

The check against e.Error on bw_RunWorkerCompleted should change for clarity:

if (e.Error == null)
{
    this.lbl_status.Text = "Good to go...";
}
else
{
    this.lbl_status.Text = "Cannot proceed, Error occured";
}

I can't probe this code right now... I hope it's ok.


Write a method to replace all spaces in a string with %20 I have implemented a solution to this Cracking the Coding Interview question:

Write a method to replace all spaces in a string with %20.
You may assume that the string has sufficient space at the end of the string to hold the additional characters,
and that you are given the true length of the string.
USE character array to perform this operation in place.

public static String conversion(String str, int length){

    char[] strChars = str.toCharArray();
    int numSpaces = 0;

    for(int i = 0; i < length; i++){
        if(strChars[i] == ' ')
            numSpaces++;
    }

    int newLength = length + 2 * numSpaces;
    char[] newChars = new char[newLength];

    for(int i = length - 1; i >= 0; i--){
        char c = strChars[i];
        if(c != ' '){
            newChars[i + 2 * numSpaces] = c;
        }
        else{
            newChars[i + 2 * numSpaces] = '0';
            newChars[i + 2 * numSpaces - 1] = '2';
            newChars[i + 2 * numSpaces - 2] = '%';
            numSpaces--;
        }
    }
    String newString = String.valueOf(newChars);
    return newString;
}

Can anyone give me any feedback on my implementation in regards of efficiency or improvements?

Your strategy of counting the spaces and then back-looping to shift the characters right (and replace spaces with %20) is good. The basic algorithm  is probably as good as it gets as a character array system.

Your variable names are decent, and the code flows well.

On the other hand, there are some small things I would change.

Possible bug

Your code, given the input conversion("abc     ", 3) you would output "abc" but you should not remove any "extra" padding in the string, you should return "abc     "

In fact, you should only really have the one char[] array. The second one is making you do bad things ;-)

Enhanced fors

Use enhanced-for loops when possible, and always use {} blocks for if-statements, even 1-liners:

for(int i = 0; i < length; i++){
    if(strChars[i] == ' ')
        numSpaces++;
}

should be:

for (char c : strChars) {
    if (c == ' ') {
        numSpaces++;
    }
}

Comments

Comment unusual loops - your second for-loop is an odd one, and it often helps the reader if you comment why a loop is non-sequential (or even if you just make sure they are aware of it).

multiple indexes

Have you considered having a separate index for each position in the array - the position of the source character, and the position of where to insert it?

public static String conversion(String str, int length){

    char[] strChars = str.toCharArray();

    int numSpaces = 0;
    for (int i = 0; i < length; i++) {
        if(strChars[i] == ' ') {
            numSpaces++;
        }
    }

    int insert = length + 2 * numSpaces - 1;

    // loop backwards through source characters.
    for (int i = length - 1; i >= 0; i--) {
        char c = strChars[i];
        if (c == ' '){
            strChars[insert--] = '0';
            strChars[insert--] = '2';
            strChars[insert--] = '%';
        } else {
            strChars[insert--] = c;
        }
    }
    return String.valueOf(strChars);
}

The benefit of two indexes is that you can keep the logic more readable ... ("more" is relative) as each index moves by one character at a time... and a space counts as 3 characters.


Simple generic auto __repr__ Preface

At some point I was tired of writing & supporting __repr__ methods, so I've decided to write it once and reuse everywhere in my classes.

Since I'm trying to write classes as simple as possible, most of them ends in something like

class MyClass:
     def __init__(self, param, other_param, ...):
         self.param = param
         self.other_param = other_param
         ...

i.e. initializer parameters names corresponds to fields names, so we can get initializer arguments by simply getattr'ing corresponding fields.

Main ideas are:

I want to have informative string which includes parameters involved in instance creation. For simple cases it should be possible to copy string & paste in some place (e.g. REPL session) and have similar object definition with as less work as possible. This helps a lot during debugging sessions, logging, especially in failed test cases with randomly generated data.

Great examples are Counter & OrderedDict from collections standard library:

>>> from collections import Counter
>>> Counter(range(2))
Counter({0: 1, 1: 1})
>>> Counter(range(2)) == Counter({0: 1, 1: 1})
True
>>> from collections import OrderedDict
>>> OrderedDict.fromkeys(range(2))
OrderedDict([(0, None), (1, None)])
>>> OrderedDict.fromkeys(range(2)) == OrderedDict([(0, None), (1, None)])
True

Once signature change, __repr__ should handle this automatically for simple cases like renaming/removing/changing order of parameters.

Attempt

Note: following code requires Python3.5+ due to yield from statement

import inspect
from collections import (OrderedDict,
                         abc)
from typing import (Any,
                    Callable,
                    Iterable,
                    TypeVar,
                    Union)

Domain = TypeVar('Domain')
Range = TypeVar('Range')
Map = Callable[[Domain], Range]

Constructor = Callable[..., Domain]
Initializer = Callable[..., None]

def generate_repr(constructor_or_initializer: Union[Constructor, Initializer],
                  *,
                  field_seeker: Callable[[Domain, str], Any] = getattr
                  ) -> Map[Domain, str]:
    signature = inspect.signature(constructor_or_initializer)
    parameters = OrderedDict(signature.parameters)
    # remove `self`
    parameters.popitem(0)
    to_positional_argument_string = repr
    to_keyword_argument_string = '{}={!r}'.format

    def __repr__(self: Domain) -> str:
        return (type(self).__qualname__
                + '(' + ', '.join(to_arguments_strings(self)) + ')')

    def to_arguments_strings(object_: Domain) -> Iterable[str]:
        for parameter_name, parameter in parameters.items():
            field = field_seeker(object_, parameter_name)
            if parameter.kind == inspect._VAR_POSITIONAL:
                if isinstance(field, abc.Iterator):
                    yield '...'
                else:
                    yield from map(to_positional_argument_string, field)
            elif parameter.kind == inspect._VAR_KEYWORD:
                yield from map(to_keyword_argument_string,
                               field.keys(), field.values())
            elif parameter.kind in {inspect._POSITIONAL_ONLY,
                                    inspect._POSITIONAL_OR_KEYWORD}:
                yield to_positional_argument_string(field)
            else:
                yield to_keyword_argument_string(parameter_name, field)

    return __repr__

Test

Let's define our class like

class A:
    def __init__(self, positional, *variadic_positional, keyword_only,
                 **variadic_keyword):
        self.positional = positional
        self.variadic_positional = variadic_positional
        self.keyword_only = keyword_only
        self.variadic_keyword = variadic_keyword

    __repr__ = generate_repr(__init__)

After that

>>> A(1, 2, 3, keyword_only='some', a={'sample': 42}, b={1, 2})
A(1, 2, 3, keyword_only='some', b={1, 2}, a={'sample': 42})

Honestly it looks like most of your code can't be simplified. However I found:

to_positional_argument_string is a mouth full, and people would understand your code easier if you just use repr.
to_keyword_argument_string is another mouth full and so you could call it kw_repr.
You may want to add a comment explaining what isinstance(field, abc.Iterator) is for. This is because at first it confused me.
[Controversial] You can change your code so that you don't need to be passed constructor_or_initializer. To do this you'd create parameters when __repr__ is first used.


A bitwise common prefix algorithm in Python Is my algorithm slow because it has problems? Or, is any bitwise common prefix solution not going to give me the performance I'm looking for?

After profiling my algorithm, I found that over 60% of the time was spent on one line len(os.path.commonprefix((item1, item2))). I'm only looking for the length of the prefix

To solve this I tried to write a bitwise prefix solution

def bit_prefix(a, b):
    min_len = min(len(a), len(b))
    if min_len > 0:
        x = str(bin(
            int(a[::-1].encode('hex'), 16) ^ int(b[::-1].encode('hex'), 16)))
        y = x.strip('0')
        if len(y) is 1:
            return min_len
        else:
            return (len(x) - len(y)) / 8
    else:
        return 0

I've only gotten a marginal improvement in speed with long prefixes

a = 'a' * 1000000 + 'z'

b = 'a' * 900000 + 'z'

timeit.timeit(lambda: bit_prefix(a, b), number=100)
Out[34]: 6.340372534867129

timeit.timeit(lambda: len(os.path.commonprefix((a, b))), number=100)
Out[35]: 7.5483549056534684

print bit_prefix(a, b), len(os.path.commonprefix((a, b)))
900000 900000

And my algorithm performs more poorly with short prefixes

a = 'aaz'

b = 'az'

timeit.timeit(lambda: bit_prefix(a, b), number=1000000)
Out[42]: 3.968956086175467

timeit.timeit(lambda: len(os.path.commonprefix((a, b))), number=1000000)
Out[43]: 1.1592788235707303

print bit_prefix(a, b), len(os.path.commonprefix((a, b)))
1 1

If my algorithm isn't broken and a bitwise solution won't give me the performance boost I'm looking for, can you refer me to a common prefix solution that would outperform os.path.commonprefix?

Here's the bit_pefix profile

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    29                                           @profile
    30                                           def bit_prefix(a, b):
    31     36140        81099      2.2      4.2      min_len = min(len(a), len(b))
    32     36140        49386      1.4      2.5      if min_len > 0:
    33     36140        49739      1.4      2.6          x = str(
    34     36140        47846      1.3      2.5              bin(
    35     36140        47232      1.3      2.4              int(
    36     36140        89499      2.5      4.6              a[::-1]
    37     36140       242994      6.7     12.5              .encode('hex'),
    38     36140       164442      4.6      8.4              16)
    39                                                       ^
    40     36140        49601      1.4      2.5              int(
    41     36140        88745      2.5      4.6              b[::-1]
    42     36140       216488      6.0     11.1              .encode('hex'),
    43     36140       504425     14.0     25.9              16)))
    44     36140       187571      5.2      9.6          y = x.strip('0')
    45     36140        61027      1.7      3.1          if len(y) is 1:
    46                                                       return min_len
    47                                                   else:
    48     36140        67507      1.9      3.5              return (len(x) - len(y)) / 8
    49                                               else:
    50                                                   return 0

Update

I've created another algorithm that seems to be faster than others. Here's the Code Review for that.

Is my algorithm slow because it has problems?

Yes. The fundamental problem is that it always processes every character of the input.

There are some obvious improvements which don't address this fundamental problem:

Since the common prefix can't be more than min_len, truncate both strings to min_len before encoding them.
bin returns a string, so str(bin(...)) is overkill.
bin(int(a[::-1].encode('hex'), 16) ^ int(b[::-1].encode('hex'), 16)) is also overkill: what you care about is the position of the lowest set bit of int(a[::-1].encode('hex'), 16) ^ int(b[::-1].encode('hex'), 16). You can extract the lowest set bit directly from the integer as

differences = int(a[::-1].encode('hex'), 16) ^ int(b[::-1].encode('hex'), 16)
least_significant_difference = differences ^ (differences - 1)

(That actually gives a binary number which is all 1s from the lowest set bit to the least significant bit).

Then you can either convert to string and find the length, or take the logarithm base 2.

But the fundamental problem is still there: the fastest code is the code which isn't executed, and when the prefix is very short then a simple loop from the start which compares characters is hard to beat. If you really have to beat it, I think you're probably looking at writing some C which coerces a pointer to (16-bit?) characters into a pointer to 64-bit integers and uses that to compare a block of characters at a time. Beware endianness.


Random Delay Generator for Workflow Foundation Problem statement: In Windows Workflow Foundation we have a long running state machine.  We had an issue with SQL server deadlocks putting workflows into a suspended state.  Since Workflow instances are independent actors, we cannot easily coordinate these calls.  As part of our workflow error handling strategy, if the offending activity receives a sql exception, we want to delay and try again and do this in a loop up to a certain number of retries.

We need a (semi) random int. Now the proper thing to do is generate one random list and use the next(min,max), but each workflow instance is an independent actor and I don't want to make a central service just for this.

My strategy, is by using the leftmost digits of DateTime->Ticks as the random seed, this will generate a random enough number for a delay activity. The plan is that eventually all workflows will advance to their next state.  Performance is not a concern. I know random is an expensive class to spin up, but I can live with it.

Example:
There are 100 workflow instances that are scheduled to change state because a new workday has started. They all want to call sql at the same time.  Workflow Instances 90,91,and 92 make calls at the same time.  90 goes through just fine, but causes a deadlock for 91 and 92.

So 91 and 92 go into a delay activity, but before we go into the delay, we need to set a somewhat random number of seconds to wait.

All Workflow Instances have a minSeconds = 10 and maxSeconds = 600

Workflow Instance 1 : never calls our random generator

Workflow Instance 2 : never calls our random generator

Workflow Instance 90: never calls our random generator

Workflow Instance 91 : hits a deadlock calls generator which returns 63

Workflow Instance 92 : hits a deadlock calls generator which returns 123

Since 91 and 92 have to wait a different number of seconds, 91 waits less time, executes, then 60 seconds later 92 wakes up from the delay and calls our large transactional stored procedure and ... no more deadlocks.

  public sealed class RandomSecondsGenerator : CodeActivity<int>
    {
        [RequiredArgument]
        public InArgument<int> MinSeconds { get; set; }

        [RequiredArgument]
        public InArgument<int> MaxSeconds { get; set; }

        /// <summary>
        /// Usage,
        /// 1. Create a workflow variable, as an int, to hold the number of seconds.
        /// 2. Create a workflow variable, as a timespan, to hold the conversion from int to timespan.
        /// Call this Activity, and set the result to the in variable
        /// Use the assignment activity to convert the int to a timespan like below
        /// timespanToWait = TimeSpan.FromSeconds(secondsToWait);
        /// Add the delay activity to the workflow ...
        /// </summary>
        /// <param name="context"></param>
        /// <returns></returns>
        protected override int Execute(CodeActivityContext context)
        {
            var ticks = DateTime.Now.Ticks;
            var intArray = ticks.ToString().Reverse().Take(8);
            var backTogether = intArray.Aggregate("", (current, item) => current + item);
            var seed = Convert.ToInt32(backTogether);
            var rand = new Random(seed);

            var min = context.GetValue(MinSeconds);
            var max = context.GetValue(MaxSeconds);
            var secondsToWait = rand.Next(min, max);
            return secondsToWait;
        }
    }
}

don't new it every time
it is expensive and is not as random as just keep using random
new Random() is based on a time seed

    private Random rand = new Random();
    protected override int Execute(CodeActivityContext context)
    {
        var min = context.GetValue(MinSeconds);
        var max = context.GetValue(MaxSeconds);
        var secondsToWait = rand.Next(min, max);
        return secondsToWait;
    }

OP said the many of the class are generated at once
Just create random once and pass it in the constructor

public class RandomSecondsGenerator
{
    public int MinSeconds { get; set; }
    public int MaxSeconds { get; set; }
    private Random rand;
    protected int Execute(DateTime context)
    {
        var secondsToWait = rand.Next(MinSeconds, MaxSeconds);
        return secondsToWait;
    }
    public RandomSecondsGenerator(Random random)
    {
        rand = random;
    }
}


Building XML by enumerating through array & hashes data =
[ { 'name' => 'category1',
    'subCategory' => [ {'name' => 'subCategory1',
                        'product' => [ {'name' => 'prodcutName1',
                                        'desc' => 'desc1'},
                                       {'name' => 'prodcutName2',
                                        'desc' => 'desc2'}]
                        } ]

  },
  {
    #category2 ...and so on
  }
]

Just recently finished a small project with Ruby. I used the above array of hashes to produce a XML.

I think my solution is quite messy nesting loops and builder tags. Can anyone help me out with a more elegant approach?

builder = Nokogiri::XML::Builder.new { |xml|
data.each { |category| # go through every category in the data array
    xml.category {
        xml.name category['name']
        category['subCategory'].each { |subCategory| # go through each subCategory in the category
            xml.subCategory {
                xml.name subCategory['name']
                subCategory['product'].each { |product| # go though each products and print their data
                    xml.name product['name']
                    xml.desc product['desc']
                }
            }
        }
    }
}

puts builder.to_xml

Here's a nice recursive solution, that creates <key>value</key> from 'key'=>'value' entries in your hash. If the value is an array, it instead recurses, using the key name as a wrapper element.

require 'nokogiri'

def process_array(label,array,xml)
  array.each do |hash|
    xml.send(label) do                 # Create an element named for the label
      hash.each do |key,value|
        if value.is_a?(Array)
          process_array(key,value,xml) # Recurse
        else
          xml.send(key,value)          # Create <key>value</key> (using variables)
        end
      end
    end
  end
end

builder = Nokogiri::XML::Builder.new do |xml|
  xml.root do                           # Wrap everything in one element.
    process_array('category',data,xml)  # Start the recursion with a custom name.
  end
end

puts builder.to_xml

When used with this data…

data = [
  { 'name' => 'category1',
    'subCategory' => [
      { 'name' => 'subCategory1',
        'product' => [
          { 'name' => 'productName1',
            'desc' => 'desc1' },
          { 'name' => 'productName2',
            'desc' => 'desc2' } ]
      } ]
  },
  { 'name' => 'category2',
    'subCategory' => [
      { 'name' => 'subCategory2.1',
        'product' => [
          { 'name' => 'productName2.1.1',
            'desc' => 'desc1' },
          { 'name' => 'productName2.1.2',
            'desc' => 'desc2' } ]
      } ]
  },
]

…you get this result:

<?xml version="1.0"?>
<root>
  <category>
    <name>category1</name>
    <subCategory>
      <name>subCategory1</name>
      <product>
        <name>productName1</name>
        <desc>desc1</desc>
      </product>
      <product>
        <name>productName2</name>
        <desc>desc2</desc>
      </product>
    </subCategory>
  </category>
  <category>
    <name>category2</name>
    <subCategory>
      <name>subCategory2.1</name>
      <product>
        <name>productName2.1.1</name>
        <desc>desc1</desc>
      </product>
      <product>
        <name>productName2.1.2</name>
        <desc>desc2</desc>
      </product>
    </subCategory>
  </category>
</root>

However, if I had control over the XML schema, I'd do this instead:

require 'nokogiri'

def process_array(label,array,xml)
  array.each do |hash|
    kids,attrs = hash.partition{ |k,v| v.is_a?(Array) }
    xml.send(label,Hash[attrs]) do
      kids.each{ |k,v| process_array(k,v,xml) }
    end
  end
end

builder = Nokogiri::XML::Builder.new do |xml|
  xml.root{ process_array('category',data,xml) }
end

puts builder.to_xml

<?xml version="1.0"?>
<root>
  <category name="category1">
    <subCategory name="subCategory1">
      <product name="productName1" desc="desc1"/>
      <product name="productName2" desc="desc2"/>
    </subCategory>
  </category>
  <category name="category2">
    <subCategory name="subCategory2.1">
      <product name="productName2.1.1" desc="desc1"/>
      <product name="productName2.1.2" desc="desc2"/>
    </subCategory>
  </category>
</root>

…but perhaps you're dealing with some terrible XML schema like PList.


Reflection optimization for export CSV on large scale So, I'm building export/import CSV helper. I have some performance issues in the code below. it takes me to parse CSV of 25,000 rows at 7 seconds.
If someone can help, it will be awesome!

public System.IO.Stream ParseContent<T>(IEnumerable<T> entities) where T : class
    {
        if (entities == null)
            throw new ArgumentException(nameof(entities), "List accepted is empty.");

        Type type = entities.First().GetType();

        PropertyInfo[] properties = type.GetProperties();

        string headers = GenerateTemplate(properties);

        //No headers accepted - cannot export the content
        if (string.IsNullOrEmpty(headers))
            return null;

        string contentToExport = $"{headers}{NewLineDelimiter}";

        foreach (T entity in entities)
        {
            if (entity == null)
                continue;

            string template = this.ExportLine(entity, properties);

            contentToExport += $"{template}{NewLineDelimiter}";
        }

        byte[] bytes = System.Text.Encoding.UTF8.GetBytes(contentToExport);
        System.IO.MemoryStream memoryStream = new System.IO.MemoryStream(bytes);

        return memoryStream;
    }

    private string ExportLine<T>(T entity, PropertyInfo[] properties) where T : class
    {
        if (entity == null || properties == null)
            return string.Empty;

        string template = "";

        foreach (PropertyInfo property in properties)
        {
            string value = null;

            if (property.PropertyType.IsGenericType && property.PropertyType.GetGenericTypeDefinition() == typeof(IEnumerable<>))
            {
                Type underlyingType = property.PropertyType.GetGenericArguments()[0];

                if (underlyingType.IsValueType || underlyingType == typeof(string))
                {
                    System.Collections.IEnumerable list = (System.Collections.IEnumerable)property.GetValue(entity);

                    value = string.Join(EnumerableValueDelimiter, list.Cast<string>());
                }
            }
            else if (property.PropertyType.IsClass && (!property.PropertyType.IsPrimitive && !property.PropertyType.IsEnum) && property.PropertyType != typeof(string))
            {
                //Object type. need to be serialized
                object propertyValue = property.GetValue(entity);

                if (propertyValue != null)
                    value = JsonConvert.SerializeObject(propertyValue);
                else
                    value = "null";
            }
            else
            {
                value = property.GetValue(entity)?.ToString();
            }

            if (string.IsNullOrEmpty(value))
                value = "";

            template += $"{value}{LineValuesDelimiter}";
        }

        //Removing the last delimiter at the row.
        if (template.Length > 0)
            template = template.Remove(template.Length - 1, 1);

        return template;
    }

ParseContent()

Type type = entities.First().GetType(); can throw an exception if entities doesn't contain any items. I may be wrong but you could use the T as well like  Type type = typeof(T);.
If entities is null an ArgumentNullException should be thrown instead of an ArgumentException.
The foreach could be simplified and you should use a StringBuilder instead of concating strings in a loop. Thats because strings are immutable and for each contentToExport += $"{template}{NewLineDelimiter}"; you create a new string object.
If the right hand side of an assignment makes the type clear one should use var instead of the concrete type.
Omitting braces {} althought they might be optional can lead to hidden and therefor hard to find bugs. I would like to encourage you to always use them.
Having a variable memoryStream doesn't buy you anything. Just return the new memorystream.

Applying these points will lead to

public System.IO.Stream ParseContent<T>(IEnumerable<T> entities) where T : class
{
    if (entities == null)
    {
        throw new ArgumentNullException(nameof(entities), "List accepted is empty.");
    }
    if (!entities.Any())
    {
        //assuming thats the desired behaviour
        return null;
    }

    Type type = typeof(T);

    PropertyInfo[] properties = type.GetProperties();

    string headers = GenerateTemplate(properties);

    //No headers accepted - cannot export the content
    if (string.IsNullOrEmpty(headers))
    {
        return null;
    }

    StringBuilder contentToExport = new StringBuilder( $"{headers}{NewLineDelimiter}");

    foreach (T entity in entities.Where(e=>e!=null))
    {

        string template = this.ExportLine(entity, properties);
        contentToExport.Append($"{template}{NewLineDelimiter}");
    }

    byte[] bytes = System.Text.Encoding.UTF8.GetBytes(contentToExport.ToString());

    return new System.IO.MemoryStream(bytes);
}


Service per Controller We have a base module consuming the company API. In each app I've written almost always a service per controller. Because most controllers are specific to a entity (a single user, a single post, list of posts written by a user), when I initialize them to query the data I have to request for that Id. So the service requires the controller to specify an Id.

Because of this I've decided the following: this services return a function which will construct the service object receiving the Id as parameter. Then the controller builds the service. With this I'm reusing those services, i.e.: A user edit his personal information on AccountController and administrator edit the personal user information on UsersListController both building the UserEditService passing the Id.

    // View User Service
function(UserResource, $log, ErrorHandler) {
    var fn = function(userId) {
       var id = userId,
           svc = {
               user: {}
           };

       svc.get = function(){
           UserResource.get(userId)
               .then(function(res)  {
                    svc.user = res.data.user;
                    svc.interests = res.data.interests;
               }, ErrorHandler.error);
       };
       return svc;
    };
    return fn;
}

// View User Controller
function(UserViewService, $scope, $stateParams) {
    var svc = new UserViewService(stateParams.id);

    this.user = svc.user;
    function watchUser() {
        return svc.user;
    };
    $scope.$watch(watchUser, function(current, previous) {
        svc.user = current;
    });
};

Am I implementing some ugly/confuse code to deal with this? I'm only doing this on applications, the base modules services and methods are encapsulated, they always receive the Id in methods parameters. But on apps, each service would had to receive the Id on each parameter in order to be reused across several controllers (they could be loaded in the same session consuming the same service).

I've tried to avoid this extra service and calling the UserResource on controller but I always end up with controller handling business operations.

"A service per controller" sounds like a bad architecture. A good service should 'know' nothing about controllers and presentation details. Controllers should be thin, and any repeating code there should be abstracted into general purpose services not specific to controllers.

"UserViewService" should never be a name of Controller! And comments are not there to fix the confusion created by bad naming! In Angular, there are Services, Controllers, and Views - these are your Angular templates. You can have UserService and UserController, then your template can be called user-view.

Having new inside your Controller looks like a "code smell" to me. Controllers are only executed once upon initialization, and their purpose is to glue your data to your view (via Angular's $scope (which is in the ViewModel layer). The data should be either ready or you should get it from a service. Any other work like object creation is bloating your controller with too many responsibilities.

Further, it is Angular's job to create Services for you, so no need for new UserViewService whatsoever.

This function naming is misleading:

function watchUser() {
    return svc.user;
};

The name should tell roughly what function does, so I can skip this function's definition and still understand the rest of the code. This is a standard "getter"-function and should be name like that:

function getUser (svc) {
    return svc.user;
}

Note. Your code looks cleaner without ; after function declarations.

You watch svc.user and then update it upon change - this looks like a loop and may lead to your browser crash. Read best practices on using Angulars watch.


OOP in PHP. Exercising on creating a blog I am attempting to do object oriented programming in right way, and appreciate any review to see if my approach is correct.

class_blogpost.php as below:

<?php
class BlogPosts {

function __construct() {
    # code...
}

public function ListAllBlogPosts() {
    $DB = DatabaseFactory::getFactory()->Connect();
    $query = $DB->prepare("SELECT * FROM posts ORDER BY date DESC");
    $query->execute();
    $list_all = $query->fetchAll(PDO::FETCH_ASSOC);
    return $list_all;
}

public function ShowABlogPost($post_id) {

    if (isset((int)$_GET["id"];)) {
        $post_id = (int)$_GET["id"];
        $DB = DatabaseFactory::getFactory()->Connect();
        $query = $DB->prepare("SELECT * FROM posts WHERE id = :post_id");
        $query->bindParam(':post_id', $post_id, PDO::PARAM_INT);
        $query->execute();
        $show_post = $query->fetchAll(PDO::FETCH_ASSOC);
        return $show_post[0];
    } else {
        echo "Post not found.";
    }

}

}

$blogposts = new BlogPosts;

Below is code for home.php which uses ListAllBlogPosts() function:

<?php
include_once 'connect.php';
$title = "Home";
include_once 'header.php';
$list_all = $blogposts->ListAllBlogPosts();
?>

<?php
foreach ($list_all as $r) {
  $id = $r["id"];
  $title = $r["title"];
  $post = $r["post"];
  $date = $r["date"];
?>

<?php echo $title; ?>

<?php echo $post; ?>

        <a href="post.php?id=<?php echo $id; ?>">more

<?php
} #ends foreach loop
?>

And finally post.php below, user reaches as post.php?id=1 like that:

<?php
include_once 'connect.php';
$id = $blogposts->ShowABlogPost($_GET["id"])["id"];
$title = $blogposts->ShowABlogPost($id)["title"];
$date = $blogposts->ShowABlogPost($id)["date"];
$post = $blogposts->ShowABlogPost($id)["post"];
include_once 'header.php';
?>

<?php echo $title; ?>

<?php echo $post; ?>

That's a good start but the structure is overall inefficient, that leads to code duplication.

the main problem is the violation of the Single responsibility principle. The ShowABlogPost does four things at a time:

connects to the database
checks the HTTP data
fetches a certain record from the database
displays some error

it should be doing only one thing, what it's intended for - fetching a record from a database.

Another issue is multiple calls to ShowABlogPost(). It would be, like, if you have to pay a cab driver 44.50 and you got this exact amount in your pocket, but would fetch the notes and coins one by one instead of taking them all at once.

You already have an array that contains all the data from the first call. So just use this array.

Other issues are:

$DB = DatabaseFactory::getFactory()->Connect(); is duplicated in every method. Why not to make a database connection a class variable? So it will be available in all class methods by default.
PDO::FETCH_ASSOC could be made the default fetch option so you won't have to call it every time explicitly
when no variables are going to be used in the query, there is no use for the prepared statements.
when you need to fetch only one row, then you have to use fetch() method, not fetchAll()

So let's rewrite your class

<?php
class BlogPosts {

    protected $db;

    function __construct($db) {
        $this->db = $db;
    }

    public function ListAllBlogPosts() {
        $sql = "SELECT * FROM posts ORDER BY date DESC";
        $return $this->db->query($sql)->fetchAll();
    }

    public function ShowABlogPost($post_id) {
        $query = $this->db->prepare("SELECT * FROM posts WHERE id = :post_id");
        $query->bindParam(':post_id', $post_id, PDO::PARAM_INT);
        $query->execute();
        return $query->fetch();
    }
}

And now we can rewrite post.php

<?php
include_once 'connect.php';
include_once 'class_blogpost.php';

$db = DatabaseFactory::getFactory()->Connect();
$blogposts = new BlogPosts($db);

if (!isset($_GET["id"]) {
    die( "Empty request");
} else {
    $post = $blogposts->ShowABlogPost($_GET["id"]);
}
?>

<?php if ($post) : ?>

<?= $post['title']; ?>

<?= $post['post']; ?>

<?php else : ?>
    Post not found.
<?php endif ?>


Current ISS latitude / longitude position I'm looking into basic API examples and I've got the following to work as I want it to. Is there a better way to determine the direction of latitude and longitude? It feels like I could do it better.

import requests
from time import strftime, localtime

response = requests.get('http://api.open-notify.org/iss-now.json')

data = response.json()
status = response.status_code

if status == 200:
    timestamp = data['timestamp']
    time = strftime('%T', localtime(timestamp))

    latitude = data['iss_position']['latitude']
    if float(latitude) > 0:
        lat_dir = 'N'
    else:
        lat_dir = 'S'

    longitude = data['iss_position']['longitude']
    if float(longitude) > 0:
        lon_dir = 'E'
    else:
        lon_dir = 'W'

    print('International Space Station position @ {}'.format(time))
    print('Latitude: {}° {}'.format(latitude, lat_dir))
    print('Longitude: {}° {}'.format(longitude, lon_dir))
else:
    print('Request unsuccessful: {}'.format(status))

Currently your code is not re-useable. To make it re-useable you should move the logic inside a function.

To do this the function we will have two functions here: _get_data_from_api and get_current_iss_location. Former gets the data from API and latter one passes that data to LatLon class and returns its instance.

The advantage of having two functions here is that the data fetching part and processing part are now isolated and each function is doing a specific thing. Plus having two isolated functions also allows us to test them easily, for example _get_data_from_api function can easily be mocked to return some data in tests and that way our tests won't rely on this external API.

For basic performance improvement we will be using a global session here to re-use previous connections.

Having a separate class LatLon to represent data allows to access the data later on and it can also have other methods that can be used later to do other stuff.

from time import strftime, localtime

import requests

session = requests.Session()

class LatLong:

    def __init__(self, latitude, longitude, timestamp):
        self.latitude = float(latitude)
        self.longitude = float(longitude)
        self.time = strftime('%T', localtime(timestamp))

    @property
    def latitude_hemishere(self):
        return 'N' if self.latitude > 0 else 'S'

    @property
    def longitude_hemisphere(self):
        return 'E' if self.longitude > 0 else 'W'

    def __str__(self):
        return (
            "<Latitude: {self.latitude}° {self.latitude_hemishere} |"
            " Longitude: {self.longitude}° {self.longitude_hemisphere} at {self.time}>").format(self=self)

    __repr__ = __str__

    def pprint(self):
        print('International Space Station position @ {}'.format(self.time))
        print('Latitude: {self.latitude}° {self.latitude_hemishere}'.format(self=self))
        print('Longitude: {self.longitude}° {self.longitude_hemisphere}'.format(self=self))

def _get_data_from_api():
    response = session.get('http://api.open-notify.org/iss-now.json')
    response.raise_for_status()
    return response.json()

def get_current_iss_location():
    data = _get_data_from_api()
    iss_position = data['iss_position']
    return LatLong(iss_position['latitude'], iss_position['longitude'], data['timestamp'])

if __name__ == '__main__':
    get_current_iss_location().pprint()

Now running the module directly will result in:

International Space Station position @ 21:14:51
Latitude: -51.518° S
Longitude: -93.6953° W

We can also import the function in other modules and get the values:

>>> loc = get_current_iss_location()
>>> loc
<Latitude: -51.616° S | Longitude: -89.694° W at 21:15:31>
>>> loc.latitude
-51.616
>>> loc.longitude
-89.694
>>> loc.time
'21:15:31'
>>> loc.pprint()
International Space Station position @ 21:15:31
Latitude: -51.616° S
Longitude: -89.694° W


Trigger custom implicit conversion on "copy" Is there anything horribly wrong with creating different representations of data behind the scenes when constructing a std::vector<T> from a std::vector<U>?

std::vector<Color> initialScene(Settings::HRes * Settings::VRes);

std::vector<Pixel> finalScene(std::begin(initialScene), std::end(initialScene));

With the Pixel(const Color&) constructor doing something along the lines of:

explicit Pixel(const Color& color) {
   // Basically swaps blue and red channels
   data[0] = color.blue;
   data[1] = color.green;
   data[2] = color.red;
}

Note: this is not exactly what the converting constructor does; it's more of an illustration.

No problem with that at all, since it's not the copy which is doing the conversion (which would be strange) but the constructor.  Since the vector templates you've mentioned are clearly of different types, the operation you describe would not at all be unexpected.  It's little different from using std::copy to read from a file into an STL container, which is a common idiom.


Let's read a random Goodreads book in an optimal way I have made the following program to gather data on random books from Goodreads, via their random books feature.

import requests
import re

URL = "https://www.goodreads.com/book/random"

while True:
    html_text = requests.get(URL).text

    #   Rating Count
    bg_rating_count     = html_text.find("", bg_rating_count)
    rating_count        = int(html_text[bg_rating_count : end_rating_count].replace(',', ''))

    if rating_count >= 30:
        if "" in html_text:

            #   Title
            bg_title    = html_text.find("<meta property=\"og:title\" content=\"") + 35
            end_title   = html_text.find("\"/>", bg_title)
            title = html_text[bg_title : end_title].replace("&amp;", '&')
            title = re.sub(r'\((.*)\)','', title)

            #   Pages
            bg_pages    = html_text.find("") + 31
            end_pages   = html_text.find(" page", bg_pages)
            pages       = int(html_text[bg_pages : end_pages])

            #   Rating
            bg_rating   = html_text.find("") + 45
            end_rating  = html_text.find("</", bg_rating)
            rating      = float(html_text[bg_rating : end_rating])

            if pages != 0:
                print(title, pages, rating, sep='\t')

This is my first time working with anything of this sort, and I am still learning Python as I go. Here's a breakdown of what the code does (or, is intended to do):

This program gets the HTML code from a random book via the URL.
Finds the book's rating count by searching for a specific HTML tag. Removes any commas in the rating number before making sure it's an integer.
If the rating count is >= 30, it accepts it and continues to gather data. Otherwise it moves on and tries another random book.
If it passes the rating count test, it checks whether it has a listed number of pages. If it does, it continues to gather data. Otherwise it moves on and tries another random book.
If it passes the page count test, it then gathers the title. If the title has an ampersand HTML code &amp; it replaces it with the actual ampersand character &. As well, if the title contains text identifying a series, it removes the series text. Here's an example of what I'm talking about, where it says (The Hunger Games #1). I am new to regex, let me know if I am doing anything horrible.
It then actually gathers the page count along with making sure it's an integer, and the book's rating with making sure it's a float.
If the page count is not 0 (which is sometimes the case for ill-detailed books), it then prints out the relevant data in a tab-separated format that I can copy-paste for later.

I understand that this program goes on forever, and I intended it like that. This program is meant to be manually used, and I halt it when I am done.

Is multi-threading a good idea? How can I improve this code, both in terms of performance, readability, or code logic?

After I've gone through your code, I decided to rewrite it because using regex to parse html isn't a good idea at all.
When you're parsing html, is recommended to use BeautifulSoup, so I'll rewrite your code using it.

First, I'll start from the imports:
from bs4 import BeautifulSoup as bs
import requests

Nothing too fancy so far, we're importing the modules we need in order to have our work done.

You had in your code a magic number (30), which we can define at the top of our program, just below the imports:
MIN_RATING_COUNT = 30

You had other numbers in your code which didn't make sense to me, so I removed them. Just let me know what their purpose was (if any) if there's any difference between my proposed solution and yours.

Moving on, we can now build a function which returns a bs object on which we can later work.
def get_html_source():
    """Docstring here."""
    html_source = requests.get(URL).text
    return bs(html_source, 'html.parser')

Now, let's build another four functions which will get us the rating count, title, pages and rating of a random book.
def get_book_rating_count(soup):
    """Docstring here."""
    return soup.find('span', attrs={'class', 'value-title'}).get_text()

def get_book_title(soup):
    """Docstring here."""
    return soup.find('h1', attrs={'class': 'bookTitle'}).get_text()

def get_book_pages(soup):
    """Docstring here."""
    return soup.find('span', attrs={'itemprop': 'numberOfPages'}).get_text()

def get_book_rating(soup):
    """Docstring here."""
    return soup.find('span', attrs={'itemprop': 'ratingValue'}).get_text()

As I was testing this out, I've noticed that book_rating_count might have different value formats so let's build another function to treat each case:
def to_float(rating_count):
    """Docstring here."""
    rating = rating_count.split()[0]
    return float(rating.replace(',', '.'))

Regarding the title, we can build another function to nicely format it: remove the newlines in it and replace &amp; with &:
def format_title(book_title):
    """Docstring here."""
    return ' '.join(book_title.split()).replace('&amp;', '&')

Last, but not least, let's build our main function:
def main():
    """Docstring here."""
    while True:
        soup = get_html_source()
        book_rating_count = get_book_rating_count(soup)

        if to_float(book_rating_count) > MIN_RATING_COUNT:
            book_pages = get_book_pages(soup)
            book_title = format_title(get_book_title(soup))
            book_rating = get_book_rating(soup)

            print('Title: {}\n'
                  'Pages: {}\n'
                  'Rating: {}\n\n'.format(book_title, book_pages, book_rating))

If you're also going to terminate the program using CTRL + C, I suggest you put your main function into a try/except block.
try:
    main()
except KeyboardInterrupt:
    print("You've decided to close the program")

The final code:
from bs4 import BeautifulSoup as bs
import requests

URL = "https://www.goodreads.com/book/random"
MIN_RATING_COUNT = 30

def to_float(rating_count):
    """Docstring here."""
    rating = rating_count.split()[0]
    if ',' in rating:
        return float(rating.replace(',', '.'))
    return float(rating)

def format_title(book_title):
    """Docstring here."""
    return ' '.join(book_title.split()).replace('&amp;', '&')

def get_html_source():
    """Docstring here."""
    html_source = requests.get(URL).text
    return bs(html_source, 'html.parser')

def get_book_rating_count(soup):
    """Docstring here."""
    return soup.find('span', attrs={'class', 'value-title'}).get_text()

def get_book_title(soup):
    """Docstring here."""
    return soup.find('h1', attrs={'class': 'bookTitle'}).get_text()

def get_book_pages(soup):
    """Docstring here."""
    return soup.find('span', attrs={'itemprop': 'numberOfPages'}).get_text()

def get_book_rating(soup):
    """Docstring here."""
    return soup.find('span', attrs={'itemprop': 'ratingValue'}).get_text()

def main():
    """Docstring here."""
    while True:
        soup = get_html_source()
        book_rating_count = get_book_rating_count(soup)

        if to_float(book_rating_count) > MIN_RATING_COUNT:
            try:
                book_pages = get_book_pages(soup)
            except AttributeError:
                book_pages = 'No pages available'
            book_title = format_title(get_book_title(soup))
            book_rating = get_book_rating(soup)

            print('Title: {}\n'
                  'Pages: {}\n'
                  'Rating: {}\n\n'.format(book_title, book_pages, book_rating))

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        print("You've decided to close the program")

If you let the above run for several minutes, you'll eventually get an AttributeError for the simple fact that some books don't have the number of pages, so I also added that into a try/except block.
Other changes that I've done:

I've added if __name__ == '__main__'
I've split the code in several functions so that each of them does its own job

Recommendations regarding your code-style:

in comments, there should be only one space after #;
there should be only one space before and after = operator;
instead of randomly using comments inside your code, just add docstrings to your functions (see PEP257 for more info);
instead of name = "He called me \"Bla\" again!" you can do name = 'He called me "Bla" again!' and vice-versa.

You can also read more about Python's style-guide, here.

Performance:
If you'll have many <30 books one ofter another, you don't have what to do but wait. As for your question regarding multithreading, threads only actually speed up I/O operations - because of the GIL.
CPU-bound operations (like the parsing/searching BeautifulSoup is doing) can't actually be done in parallel via threads, because only one thread can do CPU-based operations at a time. So you still may not see the speed up you were hoping for with this approach.
When you need to speed up CPU-bound operations in Python, you need to use multiple processes instead of threads. Luckily, you can easily do this using the multiprocessing module.


Is this PHP email validation comprehensive and complete? It might be a bit redundant, and I'm not experienced enough to wrap it all into one function. But,
I'm aiming for unicode email adresses and a quick loose but secure validation. How can I improve this?

note: I'm using PDO, so I'm mainly focusing on preventing HTML injection. Is this an issue ether way if I'm outputting user input in htmlspecialchars?

$email = "Bàt mâ'n@çupærman.dc";

function isValid($email) {
    return !preg_match('/[^ ]+@[^ ]+\.[^ ]{2,7}/', $email);
}

$encoded_email = htmlspecialchars($email, ENT_COMPAT, 'UTF-8');
if (isValid($email)) {
    echo $encoded_email. " is not valid!";
} else {
    if ($encoded_email===$email) {
    echo $encoded_email. " is valid!";
    } else {
        echo $encoded_email. " is not valid!";
    }
}

Use filters

You are a lot better off just using the built in validation filters:

$email_a = 'joe@example.com';
$email_b = 'bogus';

if (filter_var($email_a, FILTER_VALIDATE_EMAIL)) {
    echo "This (email_a) email address is considered valid.";
}
if (filter_var($email_b, FILTER_VALIDATE_EMAIL)) {
    echo "This (email_b) email address is considered valid.";
}

Handling International email addresses

International domain names are not really UTF-8 - they are punycode encoded strings (ascii). A UTF-8 local name isn't considered valid by most servers - i.e.

foo@çupærman.dc valid (after converting the domain to punycode)
foo@xn--uprman-quaf.dc valid (this is the above puny code converted)
çupærman@foo.dc INvalid

if you just want to prevent injection - run the email address through striptags before processing; you could also follow this advice and use loose validation and simply send a mail to confirm the mail exists.


QuickSort C++ Implementation using Iterators I implemented quicksort using iterators. I am will be grateful, if someone can review my piece of code. BTW, I am just selecting the last element as the pivot (which I plan to randomize later) I am mainly looking for the following:

1) Correctness of the Implementation

2) Correct usage of iterators

3)  Basic coding style.

Although above are the key things I am looking for, please feel free to point out any mistakes. Thanks!!!

#include <iostream>
#include <vector>
#include <algorithm>
#include <iterator>
#include <utility>

typedef std::vector<int>::iterator itr;

itr partition(itr left,itr right)
{

    itr i=left-1;

    itr it=left;

    while(it<right)
    {
        if(*it<=*right)
        {
            ++i;
            std::swap(*i,*it);
        }
        ++it;
    }

    std::swap(*(i+1),*right);
    return ++i;
}

void quicksort(std::vector<int>& v,itr left, itr right)
{
    if(std::distance(left,right)>=2)
    {
        itr q=partition(left,right);
        quicksort(v,left,q-1);
        quicksort(v,q+1,right);
    }
}

int main()
{
    std::vector<int> v={6,10,4,5,3,9,13,8};

    std::copy(v.begin(),v.end(),std::ostream_iterator<int>(std::cout," "));
    std::cout<<std::endl;

    quicksort(v,std::begin(v),std::end(v)-1);

    std::copy(v.begin(),v.end(),std::ostream_iterator<int>(std::cout," "));
    std::cout<<std::endl;

    return 0;
}

Templates

Sorting is pretty much the same regardless of the type of items being sorted, as long as you can compare and swap those things. Might as well implement the sorting and partitioning as templates so they can work for various containers and/or types in the containers.

Pass only what's needed

Right now, your quicksort receives the container to be sorted as a parameter. It also receives iterators referring to the beginning and end of the range to be sorted. It "uses" the container parameter only to pass it in further recursive calls--but none of the calls ever actually makes any real use of that parameter.

The parameter is never used, so you might as well not pass it.

At the same time, sorting an entire collection is common enough that it may well be worthwhile to provide that directly. For example:

// basic version that uses (only) iterators
template <class Itr>
void quicksort(Itr b, Itr e) {
    if (std::distance(b, e) > 1) {
        Itr pivot = partition(b, e);
        quicksort(b, pivot);
        quicksort(pivot, e);
    }
}

// Version to sort a whole container
template <class Container>
void quicksort(Container &c) {
    quicksort(c.begin(), c.end());
}

Don't explicitly specify std::swap

If the user has defined a swap for the type of object being sorted, you want to use it. You only want to use std::swap if they haven't defined it. To do that, you typically have a using directive to make std::swap visible in the current scope, then make an unqualified call to swap. If they've defined a swap for their type, this will find their swap via argument dependent lookup. Then, if (and only if) that fails, it will find std::swap.

if(*it<=*right)
{
    using std::swap;
    ++i;
    swap(*i,*it);
}

Decide what to support

Right now, you use std::distance to measure the distance between a pair of iterators. That lets the code work with most categories of iterators.

Unfortunately, you also use iterator + int, and that only works with random access iterators, so the code overall only works with random access iterators.

If you don't mind requiring random access iterators, then you might as well depend on them throughout, and just subtract iterators to get a distance.

Conversely, if you really want the code to work with other iterators, you should truly support that, such as by using std::advance or std::next to modify the pointers.

Minimize requirements on the contained type

Quicksort can be written so the only comparison it depends upon is operator<. Right now you use <= for no particularly good reason that I can see.


Generate regular expression based on inputted number This is my first PHP script (although I like to think I'm a good general coder).

The script accepts a number from the user, uses it to generate a regular expression, and then returns all dictionary words that match that expression.

It's currently live here, and I'd appreciate any comments on style, security, best practices, and even efficiency. Please ignore the Html for now - it's got a WordPress template to be inserted into later on.

<title>Major Analysis Number Encoder</title>

Tool for finding keywords for major analysis

    This tool uses the numbering system shown by Derren Brown in the
    excellent book
    Trick
        of the mind. For a (much better) tool that uses the older
    numbering system see
    http://www.rememberg.com/

    <form action="majorEncoder.php" method="post">
        Number to encode (at least three digits
            please): <input type="text" id="targetNumber"
            name="targetNumber" />
 <input type="submit"
            value="Generate results" name="submit" />
    </form>
<?php
if ($_POST != null) {

    $target = $_POST ['targetNumber'];
    if (strlen ( $target ) >= 3) {
        $f = '[waeiouyc]?';
        $regex = '/^' . $f . $target;
        echo 'You entered the target' . $target . '

';
        $regex = str_replace ( '1', 'l' . $f, $regex );
        $regex = str_replace ( '2', 'n' . $f, $regex );
        $regex = str_replace ( '3', 'm' . $f, $regex );
        $regex = str_replace ( '4', 'r' . $f, $regex );
        $regex = str_replace ( '5', '[fv]' . $f, $regex );
        $regex = str_replace ( '6', '[bp]' . $f, $regex );
        $regex = str_replace ( '7', 't' . $f, $regex );
        $regex = str_replace ( '8', '[sc]h' . $f, $regex );
        $regex = str_replace ( '9', '[gd]' . $f, $regex );
        $regex = str_replace ( '0', '[sz]' . $f, $regex );
        $lines = file ( 'words.txt' );

        $fl_array = preg_grep ( $regex . '$/', $lines );
        if (sizeof ( $fl_array ) > 0) {
            echo 'Exact matches:
';
            foreach ( $fl_array as $element ) {
                echo $element . '
';
            }
            echo '
';
        }
        $fl_array = preg_grep ( $regex . '/', $lines );
        if (sizeof ( $fl_array ) > 0) {
            echo 'Matches that allow trailing characters:
';
            foreach ( $fl_array as $element ) {
                echo $element . '
';
            }
        } else {
            echo "No matches found for the input $target";
        }
    } else {
        echo 'Please enter at least three digits
';
    }
}
?>

One thing I would do is that str_replace can receive an array of items to look for and an array of its replacement:

PHP str_replace

// Provides: You should eat pizza, beer, and ice cream every day
$phrase  = "You should eat fruits, vegetables, and fiber every day.";
$healthy = array("fruits", "vegetables", "fiber");
$yummy   = array("pizza", "beer", "ice cream");

$newphrase = str_replace($healthy, $yummy, $phrase);

instead of going line by line to do it.

Just nit-picking but

sizeof() is an alias of count() and count() is more widely used, so use something that is more general in the PHP community

Also $_POST != null would typically be !isset($_POST) rather as explained in this post


Create pairs of items with similarity constraints I want to create N unordered pairs of items (no repeat pairs, no pairs with repeated items), where each item is only used twice. For example, given the minimal amount of integers greater than 0, one possible set of 4 items would be:

[(0, 1), (0, 2), (3, 1), (3, 2)]

Another valid example would be:

[(0, 1), (1, 2), (2, 3), (3, 0)]

This would be invalid due to repeated pair:

[(0, 1), (0, 1), (2, 3), (3, 1)]

This would be invalid due to the item 0 being used three times:

[(0, 1), (0, 2), (0, 3), (3, 2)]

Also, to be clear the pair (0, 0) is invalid because it is a pair with a repeated item.

My current code for accomplishing this is clumsy and only works multiples of 4. For now, let's assume the code only needs to work for multiples of 4. How could I do this with a better iterator?

num_set = range(4)
res = []
for n1, n2, n3, n4 in zip(num_set[::4], num_set[3::4], num_set[1::4], num_set[2::4]):
    res.append((n1, n3))
    res.append((n1, n4))
    res.append((n2, n3))
    res.append((n2, n4))

print(res)

I feel like I should be able to use itertools.combinations(range(4), 2) and some filtering to accomplish this, but I'm not sure how.

The simplest answer I could come up with exploits the structure of the problem and gives a deterministic answer, but since I didn't specify the answer had to be random, I think it's valid.

Basically, the pairs [(0, 1), (1, 2), (2, 3), (3, 0)] are equivalent to some creative slicing of the list d_list = [0, 0, 1, 1, 2, 2, 3, 3]. I couldn't figure out how to slice it properly, so instead I used a hacky re-arrangement of d_list.append(d_list.pop(0)). This gives d_list = [0, 1, 1, 2, 2, 3, 3, 0] which I can then slice into pairs trivially.

Complete code below:

n_items = 5

nums = list(range(n_items))

d_list = []
for nn in nums:
    d_list.append(nn)
    d_list.append(nn)

d_list.append(d_list.pop(0))

pairs = []
for i1, i2 in zip(d_list[::2], d_list[1::2]):
    pairs.append((i1, i2))

It's definitely not the prettiest code, but it works.


Checking mobile SIMs Can someone improve this code? I need to eliminate the for loop.

public bool CheckMobileSim(List<Mobile_Range> numberRange, string MobileNumber)
{
    bool SimType = new bool();
    string NineDigits = MobileNumber.Substring(0, 9).ToString();
    long Number = Convert.ToInt64(NineDigits);
    if (numberRange != null)
    {
        if (numberRange.Count > 0)
        {
            for (int i = 0; i < numberRange.Count; i++)
            {
                if ((Number >= numberRange[i].RangeStart && Number < numberRange[i].RangeEnd))
                {
                    SimType = true;
                    break;
                }

            }
        }
    }

    return SimType;
}

Refactoring step 1, remove unnecessary variables and braces and return early

public bool CheckMobileSim(List<Mobile_Range> numberRange, string MobileNumber)
{
    if(numberRange == null)
        return;
    string NineDigits = MobileNumber.Substring(0, 9).ToString();
    long Number = Convert.ToInt64(NineDigits);
    for (int i = 0; i < numberRange.Count; i++)
        if ((Number >= numberRange[i].RangeStart && Number < numberRange[i].RangeEnd))
            return true;

    return false;
}

I realize that breaks with the "single point of return" wisdom but I'm not a big fan. When you have small functions like this one, it doesn't make all that much sense.

Refactoring step 2, with that cleaned up it's easy to see how this fits into a simple LINQ query

public bool CheckMobileSim(IEnumerable<Mobile_Range> numberRange, string mobileNumber)
{
    if(numberRange == null || String.IsNullOrWhitespace(mobleNumber))
        return false;

    var nineDigits = mobileNumber.Substring(0, 9);
    var number = Convert.ToInt64(nineDigits);

    return numberRange.Any(n => number >= n.RangeStart && number < n.RangeEnd);
}

Since this is a public method I added some checks and downcast List to IEnumerable, which is a looser contract and all you really need here.

By the way, .Net naming conventions are pascalCase for private and local variables CamelCase for public and protected. It's rare to use underscores.


Compare neighbors in array I have a a python function for taking in a 2D numpy array and checking if each element is the same as its neighbor elements. I feel like there's a more efficient way to do this but I'm not sure. Here is the code:

import numpy as np

def compare_neighbors(arr):
    '''
    Checks if element (i,j) is different than (i-1,j),(i+1,j),(i,j-1), or
    (i,j+1).

    --Input--
    arr: (2D np.array) array to compare all elements of

    --Returns--
    comp_arr: (2D bool np.array) bool array with the resulting comparisons.
           True means the original element is the same as its neighbors,
           False means it was different than at least neighbor
    '''
    comp_arr = np.full(arr.shape, False, dtype=bool) #initialize
    arr_height = arr.shape[0]
    arr_width = arr.shape[1]

    for i in range(arr_height): #Row
        for j in range(arr_width): #column

            center = arr[i,j]

            #Check edges
            if i == 0: #left side
                left = arr[i,j]
            else:
                left = arr[i-1, j]

            if i == arr_height - 1: #right side
                right = arr[i,j]
            else:
                right = arr[i+1,j]

            if j == 0: #up
                up = arr[i,j]
            else:
                up = arr[i, j-1]

            if j == arr_width - 1: #down
                down = arr[i,j]
            else:
                down = arr[i, j+1]

            comp_arr[i,j] = len(set([left, right, up, down, center])) == 1
    return comp_arr

If it is helpful, here are the tests I used for testing it:

A = np.array([[1,1],
              [1,1]])

comp_arr_A = compare_neighbors(A)

B = np.array([[2,2],
              [2,2]])

comp_arr_B = compare_neighbors(B)

C = np.array([[1,1,1,1,1,1,1,1,1],
              [1,1,1,1,1,1,1,1,1],
              [1,2,2,2,2,2,2,2,1],
              [1,2,2,1,1,1,2,2,1],
              [1,2,2,2,2,2,2,2,1],
              [1,1,1,1,1,1,1,1,1]])

comp_arr_C = compare_neighbors(C)

D = np.array([[1,1,1],
              [1,2,1],
              [1,1,1]])

comp_arr_D = compare_neighbors(D)

print(A)
print()
print(comp_arr_A)
print()
print(B)
print()
print(comp_arr_B)
print()
print(C)
print()
print(comp_arr_C)
print()
print(D)
print()
print(comp_arr_D)

which returns

[[1 1]
 [1 1]]

[[ True  True]
 [ True  True]]

[[2 2]
 [2 2]]

[[ True  True]
 [ True  True]]

[[1 1 1 1 1 1 1 1 1]
 [1 1 1 1 1 1 1 1 1]
 [1 2 2 2 2 2 2 2 1]
 [1 2 2 1 1 1 2 2 1]
 [1 2 2 2 2 2 2 2 1]
 [1 1 1 1 1 1 1 1 1]]

[[ True  True  True  True  True  True  True  True  True]
 [ True False False False False False False False  True]
 [False False False False False False False False False]
 [False False False False False False False False False]
 [False False False False False False False False False]
 [ True False False False False False False False  True]]

[[1 1 1]
 [1 2 1]
 [1 1 1]]

[[ True False  True]
 [False False False]
 [ True False  True]]

as expected. All I need the function to do is check each element and compare it to its left, right, above, and bellow neighbors. If it is the same as them then the compare_array in that index is True and if it is different than any of its neighbors then False.

You can improve iterating over the array by using np.ndenumerate to get the current coordinates and current item. From the coordinates you can derive the neighbouring elements.
You can also use inverse checking to only set the respective field to False if a neighbor does not match:

from contextlib import suppress

def compare_neighbors(arr):

    comp_arr = np.full(arr.shape, True, dtype=bool)

    for (x, y), item in np.ndenumerate(arr):
        # Check left.
        if x >= 0:
            if arr[x-1, y] != item:
                comp_arr[x, y] = False
                continue

        # Check right.
        with suppress(IndexError):
            if arr[x+1, y] != item:
                comp_arr[x, y] = False
                continue

        # Check top.
        with suppress(IndexError):
            if arr[x, y+1] != item:
                comp_arr[x, y] = False
                continue

        # Check bottom.
        if y >= 0:
            if arr[x, y-1] != item:
                comp_arr[x, y] = False
                continue

    return comp_arr


Custom route for writing friendly URLs in ASP.NET MVC 3 I'm trying to build friendly URLs like /post/1/my-first-post. I started out with building my links like this:

@Html.ActionLink(Model.BlogPost.Title, "Index", "Post", new
{
   id = Model.BlogPost.Id,
   title = Model.FriendlyUrl
},
new { })

which were processed by this route:

routes.MapRoute(
   "Post", // Route name
   "posts/{id}/{title}", // URL with parameters
    new { controller = "Post", action = "Index" }, // Parameter defaults
    new { id = "^[0-9]+$" }
 );

When I create my view model, I would give it an IUrlNiceName which it used to create the output of FriendlyUrl:

public string FriendlyUrl
{
    get{ return UrlNiceName.ConvertToNiceName(BlogPost.Title); }
}

Then I felt like I didn't want to manage UrlNiceName-ing on the controller level so then I build a custom Route to do this for me:

public class NiceNameRoute : RouteBase
{
    private IUrlNiceNames UrlNiceNames { get; set; }
    private string UrlPrefix { get { return "post/"; } }

    public NiceNameRoute(IUrlNiceNames urlNiceNames)
    {
        UrlNiceNames = urlNiceNames;
    }

    public override RouteData GetRouteData(HttpContextBase httpContext)
    {
        RouteData result = null;
        string requestedURL = httpContext.Request.AppRelativeCurrentExecutionFilePath;

        //match e.g. post/1 or post/1/my-first-post
        Regex regex = new Regex(UrlPrefix + "[0-9]+($|/)");

        if (regex.Match(requestedURL).Success)
        {
            //get the start of the post id
            int prefixPos = requestedURL.IndexOf(UrlPrefix, StringComparison.OrdinalIgnoreCase);
            int startIdPos = prefixPos + UrlPrefix.Length;

            //get the end of the post id
            int endIdPos = requestedURL.IndexOf("/", startIdPos, StringComparison.OrdinalIgnoreCase);
            endIdPos = endIdPos == -1 ? requestedURL.Length : endIdPos;

            //get post id
            string id = requestedURL.Substring(startIdPos, endIdPos - startIdPos);

            result = new RouteData(this, new MvcRouteHandler());
            result.Values.Add("controller", "Post");
            result.Values.Add("action", "Index");
            result.Values.Add("id", id);
        }

        return result;
    }

    public override VirtualPathData GetVirtualPath(RequestContext requestContext,
        RouteValueDictionary values)
    {
        VirtualPathData result = null;

        if (values.ContainsKey("title")
            && Convert.ToString(values["action"]) == "Index"
            && Convert.ToString(values["controller"]) == "Post")
        {
            string niceName = UrlNiceNames.ConvertToNiceName(Convert.ToString(values["title"]));

            result = new VirtualPathData(this, UrlPrefix + Convert.ToString(values["id"]) + "/" + niceName);
        }
        return result;
    }
}

which I registered like this:

routes.Add(new NiceNameRoute(new HyphenUrlNiceNames()));

and generate links like this:

@Html.ActionLink(Model.BlogPost.Title, "Index", "Post", new
{
   id = Model.BlogPost.Id,
   title = Model.BlogPost.Title
},
new { })

I first wanted to take a crack at this but now I want to know how industry experts do it. Am I over engineering? Is there a simpler approach that is also elegant? Are there any issues with my custom Route?

Personally I think you are over thinking it. Why not just create a string extension method called FriendlyString or PrepareForUrl and then use the original route

@Html.ActionLink(Model.BlogPost.Title, "Index", "Post", new
{
   id = Model.BlogPost.Id,
   title = Model.BlogPost.Title.FriendlyString()
}, null)


Conditional string building in Ruby I can see a possible performance optimization with declaring 2 arrays and iterating the $fake_widgets only once, but how else can I  make the following conditional string building more elegant in Ruby?

pending_coloring = $fake_widgets.select do |widget|
  widget[:status] == 'awaiting_coloring' && widget[:owner_color] == @owner.id
end

pending_processing = $fake_widgets.select do |widget|
  widget[:status] == 'awaiting_processing' && widget[:owner_color] == @owner.id
end

notification_text = "You have "

if pending_coloring.size == 1
  notification_text += "#{pending_coloring.size} widget to color"
elsif pending_coloring.size > 1
  notification_text += "#{pending_coloring.size} widgets to color"
end

if pending_coloring.size > 0 && pending_processing.size > 0
  notification_text += " and "
end

if pending_processing.size == 1
  notification_text += "#{pending_processing.size} widget to process"
elsif pending_processing.size > 1
  notification_text += "#{pending_processing.size} widgets to process"
end

notification_text += ". Go to widget dashboard"

First of all, you have weird behaviour if pending_coloring and pending_processing are both empty arrays.  The output would then be

You have . Go to widget dashboard

I assume that in such a case, it should just say "Go to widget dashboard".

I think that pluralization ought to be treated as an i18n issue.  Even if your application happens to be English-only for now, it's actually easier to generalize the solution, since pluralization is considered a solved problem in internationalization.  In particular, you can use the i18n gem (which is included in Rails, and is a simple installation with no dependencies even if you aren't using Rails).

require 'i18n'

pending_coloring = …
pending_processing = …

I18n.backend.store_translations :en, 'widget' => {
    :zero => nil,
    :one => '1 widget to %{verb}',
    :other => '%{count} widgets to %{verb}'
}
coloring_text = I18n.t 'widget', :count => pending_coloring.size, :verb => 'color'
processing_text = I18n.t 'widget', :count => pending_processing.size, :verb => 'process'

notification_text = if (!coloring_text && !processing_text) then
  ''
else
  "You have #{[coloring_text, processing_text].compact.join(' and ')}.  "
end + "Go to widget dashboard"


JavaScript solution for diagonal difference hackerrank.com - diagonal difference:

  Problem Statement

  You are given a square matrix of size N×N. Calculate the absolute
  difference of the sums across the two main diagonals.

  Input Format

  The first line contains a single integer N. The next N lines contain N
  integers (each) describing the matrix.

  Constraints

  \$1 \le N \le 100\$
  \$−100 \le A[i] \le 100\$

  Output Format

  Output a single integer equal to the absolute difference in the sums
  across the diagonals.

  Sample Input

3
11 2 4
4 5 6
10 8 -12

  Sample Output

15

  Explanation

  The first diagonal of the matrix is:

11
    5
        -12

  Sum across the first diagonal = 11+5-12= 4

  The second diagonal of the matrix is:

        4
    5
10

  Sum across the second diagonal = 4+5+10 = 19

  Difference: |4-19| =15

How can this solution be improved?

function processData(input) {
    inputAll = input.split("\n");
    var currentLine = 0;
    var numberOfInputs = parseInt(inputAll[currentLine].trim(), 10);

    currentLine = 1;
    var sumFirstDiagonal = 0;
    var sumSecondDiagonal = 0;
    var difference = 0;
    for(currentLine = 1 ; currentLine <= numberOfInputs ; currentLine++)
        {
            var inputs = inputAll[currentLine];
            var inputArray = inputs.split(" ");

            for(var i=0 ; i<=(numberOfInputs-1);i++)
                {
                    if(i == (currentLine - 1))
                        {

                            sumFirstDiagonal = sumFirstDiagonal + parseInt(inputArray[i]);
                        }
                }

            for(var j=(numberOfInputs-1) ; j>=0;j--)
                {
                    if(j ==  ( (numberOfInputs - 1) - (currentLine - 1)))
                        {

                            sumSecondDiagonal = sumSecondDiagonal + parseInt(inputArray[j]);
                        }
                }

        }
    difference = sumFirstDiagonal - sumSecondDiagonal;
    difference = Math.abs(difference);
    process.stdout.write(""+difference+"\n");
}

You don't need the inner for loops because you know exactly which array element you want for each sum:

for(currentLine = 1 ; currentLine <= numberOfInputs ; currentLine++)
    {
        var inputs = inputAll[currentLine];
        var inputArray = inputs.split(" ");

        sumFirstDiagonal += parseInt( inputArray[ currentLine - 1 ])
        sumSecondDiagonal += parseInt (inputArray [ inputArray.length - currentLine ])

    }


Removing duplicates from list and filtering non-letter characters I'm just starting out in Haskell and have been set an assignment for uni where I have to create a reverse index of words in a text file and find what line numbers they appear on. I also have to remove any stop words like "the" or "and" that aren't needed.

So far I have this, but I feel that there should be a better way to remove duplicates from a list and to filter by non-letters:

module Main where
import qualified Data.List as List
import qualified Data.Char as Char
import qualified Data.Set as Set
import qualified Data.Maybe as Maybe

main = do
    -- load file to be indexed and stop words
    text <- readFile "test.txt"
    stopWords <- readFile "stopWords.txt"

    -- convert all chars to lower case, split by lines, then split by words
    let word = map words (lines (map Char.toLower text))
    let stopWord = map words (lines (map Char.toLower stopWords))

    -- concatenate lists, remove non letters, then remove any empty list elements
    let word' = filter (not.null) (map (filter Char.isLetter) (concat word))
    let stopWord' = filter (not.null) (map (filter Char.isLetter) (concat stopWord))

    -- cast both lists to a set to remove duplicates and remove any instances of stopWord' from word'
    let index = Set.toList (Set.difference (Set.fromList word') (Set.fromList stopWord'))

    -- get line numbers from indexed words
    let lineNumbers = generateIndex index word

    -- zip index of words with line numbers and print
    mapM print (List.zip index lineNumbers)

generateIndex :: [String] -> [[String]] -> [[Int]]
generateIndex xs y = foldr (\ x -> (++) [getLineNumbers x y]) [] xs

getLineNumbers' :: String -> [[String]] -> Int -> [Int]
getLineNumbers' x [] n = []
getLineNumbers' x (y:ys) n
    | x `elem` y = n : getLineNumbers' x ys (n + 1)
    | otherwise = getLineNumbers' x ys (n + 1)

getLineNumbers :: String -> [[String]] -> [Int]
getLineNumbers x y = getLineNumbers' x y 1

Think carefully about what data structure you're using, lists aren't always the best choice. In this case you need to do two major things with your words.

Associate words with the line number they first appear on.
Remove words from one group using entries from another.

Right off the bat whenever you need to associate one set of elements with another, you should think of a map. In Haskell, you'll want to look at the documentation for Data.Map.

By reordering and reconsidering the operations you're performing on lists and sets, you should be able to rewrite your code to use maps and significantly cut down the amount of code you have written and end up with something more expressive too.

As it isn't clear that you've actually turned in this assignment I'm afraid I'll have to be coy, but here's the most pointed help I can give you on your specific questions.

Regarding removing duplicate words, the trick is to rely on the fact that maps can only contain a given key once. Pay close attention to the various ways of constructing a list that are specified in the documentation for Data.Map.

Filtering out the non-letters in a string doesn't get much simpler than filter Char.isLetter in your code above. The dissatisfaction you are feeling might be because of the gross duplication between the definitions of word and stopWord, and word' and stopWord'. Any time you have two lines in Haskell that are identical but for the value you're operating on, your code is crying out for a new function definition.

Your code is lingering in IO far more than is necessary, pull the pure data manipulating machinery out into separate functions. To start you off, knowing no more than the type signatures I've given here, what do you think these functions do? What part of your code do they replace? Take a shot at defining them.

normalize :: String -> String
??? :: [a] -> [(a, Int)]
??? :: String -> [([String], Int)]


Exercising in copy constructors and assignment operators I've just created an implementation of array to train myself in copy constructors, assignment operators, and exception safety.  Please look at this code and tell me what is wrong here.

class DumbArray {
public:
  DumbArray(size_t sz)
    : _size(sz), _buf(new int[_size])
  {
    memset(_buf, 0, _size * sizeof(int));
  }

  DumbArray(const DumbArray& arr)
    : DumbArray(arr._size)
  {
    int* newMem = new int[arr._size];

    if(newMem) {
      std::copy(arr._buf, arr._buf + arr._size, newMem);
      delete [] _buf;
      _buf = newMem;
      _size = arr._size;
    }
  }

  DumbArray(DumbArray&& arr) {
    swap(arr);
  }

  DumbArray(std::initializer_list<int> ilist)
    : DumbArray(ilist.size())
  {
    std::copy(ilist.begin(), ilist.end(), _buf);
  }

  DumbArray& operator=(const DumbArray& arr) {
    if(this != &arr) {
      DumbArray tmp(arr);
      swap(tmp);
    }

    return *this;
  }

  DumbArray& operator=(DumbArray&& arr) {
    swap(arr);
    return *this;
  }

  ~DumbArray() {
    // _size = 0;   // should this be done? object is being destroyed anyway, yup?
    delete [] _buf;
  }

  void swap(DumbArray& arr) {
    std::swap(_buf, arr._buf);
    std::swap(_size, arr._size);
  }

  int& operator[](int idx) {
    if(idx < 0 || idx >= _size) {
      throw std::out_of_range("Incorrect index");
    }

    return _buf[idx];
  }

  size_t size() const {
    return _size;
  }

private:
  size_t _size;
  int* _buf;
};

This is reasonably clean and reasonably complete code but I see some things that might improve your code.

Consider making this a template

There is no reason that this data structure should be limited to just storing int values.  Consider reworking it as a templated class to make it more general and to get additional practice.

Implement member functions non-inline

All of those inline functions can increase the size of your code.  If you declare them first and then have out-of-line implementations(at least for the non-trivial functions), it has advantages, especially if you decide to make this a templated class.

Consider making the destructor virtual

By having a concrete destructor rather than a virtual one, you are effectively prohibiting derived classes.  If that's not your intent, making your destructor virtual will make it possible to derive classes from this one.

List the required #include files

By my reckoning, this code requires the following to work:

#include <cstring>
#include <initializer_list>
#include <iterator>
#include <stdexcept>

These are therefore an essential part of the class interface.

Reconsider the use of size_t

Although it's used often in the C standard library, you don't really need it here, and requiring a C-flavored include file doesn't help much in keeping the interface small.  You could use unsigned instead and reduce the number of required include files.

Avoid comparing signed to unsigned

In your operator[], idx is declared as int but _size is size_t so the range check at the top of the function compares signed and unsigned numbers without a cast. Since negative numbers are already eliminated by the first clause, so you could use either a C-style cast (gasp!) or a reinterpret_cast on idx in the second clause.

Add a default constructor

Because DumbArray has no default constructor (one with no arguments), code like this:

std::vector<DumbArray> dv(10);  // make ten arrays

cannot compile or run.  You could either add a default value for your existing constructor or create a new default constructor.

Consider using std::fill_n instead of memset

Instead of using the old-school

memset(_buf, 0, _size * sizeof(int));

you might consider instead using fill_n:

std::fill_n(_buf, _size, 0);

since you're already using the similar std::copy in one of the constructors.

Add a const version of operator[]

As it is currently written, there's no way to index into a const DumbArray.  Adding const to operator[] will fix that problem but make sure it returns a copy of an int rather than a reference to one.