Trending Machine Learning topics for Alexa I've created a small Flash Briefing Alexa skill that reports the top 5 trending topics in Machine Learning from the My Bridge service.

The feed for the Flash Briefing skill can be pointed to an external endpoint, but the response has to be in a certain format.

Sample:

[
    {
        mainText: "Posted by Hui Fang, Software Engineer, Machine Perception Machine learning (ML) excels in many areas with well defined goals. Tasks where ...",
        redirectionUrl: "https://research.googleblog.com/2017/07/using-deep-learning-to-create.html",
        titleText: "Using Deep Learning to Create Professional-Level Photographs",
        uid: "200561",
        updateDate: "2017-07-13T20:11:59.000000Z"
    },
    ...
]

To expose an endpoint I've created a small Flask app (deployed to Heroku, runs on gunicorn), that interacts with a My Bridge API via the requests module:

from datetime import datetime
import logging

from flask import Flask, jsonify
import requests

ML_ID_URL = "https://mybridge-backend.herokuapp.com/api/v1/skills/mapping/machine-learning"
ML_TOP_URL = "https://mybridge-backend.herokuapp.com/api/v1/knowledge/skills/{id}?offset=0&limit=5&sort="

logger = logging.getLogger()
logger.setLevel(logging.INFO)

app = Flask(__name__)

@app.route("/")
def main():
    with requests.Session() as session:
        session.headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"}

        # get topic ID
        response = session.get(ML_ID_URL)
        data = response.json()

        ml_topic_id = data["body"]["id"]

        # get top 5 trending topics
        response = session.get(ML_TOP_URL.format(id=ml_topic_id))
        data = response.json()

        return jsonify([
            {
                "uid": str(topic["id"]),
                "updateDate": datetime.strptime(topic["reg_date"], "%Y-%m-%d %H:%M:%S.%f").strftime("%Y-%m-%dT%H:%M:%S.%fZ"),  # "2016-04-10T00:00:00.0Z"
                "titleText": topic["title"],
                "mainText": topic["excerpt"],
                "redirectionUrl": topic["externalURL"]
            }
            for topic in data['body']
        ])

if __name__ == '__main__':
    app.run(debug=True, use_reloader=True)

What would you improve code style and code organization wise?

Structure

Nothing! The code seems fairly straightforward and clear, and I was able to understand what you’re doing even if I didn't study flask too much. So I don’t think there’s anything wrong with the structure.

Code style

One small nitpick: some of your lines are too long (>79 characters). From PEP8:

  Limit all lines to a maximum of 79 characters.

Anyway, as a personal preference, I like to stick to PyCharm's recommendation and keep it < 120. But even with that in mind, you still have one line which doesn't fit.

Do this instead (for <120):

session.headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/59.0.3071.115 Safari/537.36"
}

Or:

session.headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5)"
                      "AppleWebKit/537.36 (KHTML, like Gecko) "
                      "Chrome/59.0.3071.115 Safari/537.36"
    }

Anyway, that's really subjective and it didn't bother me that you kept the entire User-Agent on the same line.

The same observation is also applicable for updateDate.

Flask-RESTful API

Apparently, you're trying to create some sort of RESTful API. Luckily, flask let's you do this by installing their Flask-RESTful API which you can do by running pip install flask-restful.

Flask-RESTful is a powerful extension for Flask that was designed to help you build complex RESTful APIs. It follows similar design principles to Flask so it is really easy (IMO) to pick up. Now, you might wonder why you should use this: well, your code was quite straightforward to implement but as the API grows in complexity it is easy to see this code becoming very complicated, very quickly!


How do I structure my entities for easier persistence? I am learning ORM using Hibernate by creating a movie rental application. Currently, I have the following three entities for persistence:

User
Movie
Rental (composite element for many-to-many mapping between above two)

Now, to handle the cart updates, I have the following code in a session scoped bean:

@ManagedBean
@SessionScoped
public class UserManager extends BaseBean implements Serializable
{
   ....

   private Map<Integer, Movie> cart;

   ....

   public void updateCart(Movie selectedMovie)
   {
      if (!isLoggedIn()) {
         return;
      }

      int id = selectedMovie.getMovieID();

      if (cart.containsKey(id)) {
         cart.remove(id);
      }
      else {
         cart.put(id, selectedMovie);
      }
   }
}

Now for user's purchase scenario, I need to keep track of the number of rentals per movie. So, I thought of adding that property to the Rental entity and use it as a value for the map. But, this class itself contains Movie as a property for the mapping as below:

<set name="rentals" cascade="all" lazy="true" table="USER_MOVIES">
      <key column="userRegistrationID" not-null="true"/>
      <composite-element class="Rental">
         <property name="bookingDate" column="bookingDate" type="date"/>
         <many-to-one name="movie" column="bookedMovieID" class="Movie" not-null="true"/>
      </composite-element>
</set>

So, while adding to / removing from cart, I will have to create / destroy an instance of the Rental class, which would be a bad idea. So I am thinking that I should restructure my entities.

Am I right in assuming the above? Is there a better approach for the above?

I am using the following:

JSF 2.1 + PrimeFaces
Hibernate 4.3
MySQL

You are not correct in assuming the above... If you want to count the number of rentals for one Movie you can do the following (given you are not removing records from your database):

@NamedQueries{
   @NamedQuery(name="Movie.getRentalCount",
       query="SELECT COUNT FROM Rentals r WHERE r.bookMovieId = :movieId")
}
@Entity
class Movie{

   @Transient
   public static final String getRentalCount = "uniqueQueryNameForMovie";

   // ...
}

This allows you to do the following in your purchase process:

@Inject
EntityManager entityManager;

public Long getRentalCountForMovieId(long movieId){
    TypedQuery<Long> q = entityManager.createNamedQuery(Movie.getRentalCount, Long);
    q.setParameter("movieId", movieId);
    return q.getSingleResult();
}

You'd need to use the wrapper class, as TypedQuery only accepts Objects as type.

Also it should be unneccesary to create Rental instances while changing the contents of your cart. They should only be created when you check the cart out in the end, because before that there is no need for a Rental instance...

If you decided to destroy the instances of Rental on return of the Movie: You should keep a record of older Rentals for historization and usability reasons.


Faster algorithm to tailor given mathematical expression Is there a more optimized solution to solve the stated problem?

Given an array 'arr' of 'N' elements and a number 'M', find the least index 'z' at which the equation gets satisfied. [ ] is considered as floor().

Code:

counts=0
ans=0
while(ans==0):
    s=0
    for i in range(counts,len(arr)):
        s+=int(arr[i]/(i+1-counts))
        if(s>M):
            break
    if((i+1)==len(arr) and s<=M):
        print(counts)
        ans=1
    counts+=1

Explanation:

Check array from left to right. The first index that satisfies the condition is the answer. This is more optimized than considering from right to left.
If at any time during the calculation, 's' is deemed more than M, break the loop and consider the next. This is more optimized than calculating 's' completely.

Example:

INPUT:

N=3 M=3

arr=[1 2 3]

OUTPUT:

0

This would give the answer 0 since the 0th index contains the first element to satisfy the given relation.

Thanks in advance.

To start, let's fix the PEP8 style errors using any PEP8 checker:

counts = 0
ans = 0
while(ans == 0):
    s = 0
    for i in range(counts, len(arr)):
        s += int(arr[i]/(i+1-counts))
        if(s > M):
            break
    if((i+1) == len(arr) and s <= M):
        print(counts)
        ans = 1
    counts += 1

The parentheses around the conditions are also not usual Python style:

counts = 0
ans = 0
while ans == 0:
    s = 0
    for i in range(counts, len(arr)):
        s += int(arr[i]/(i+1-counts))
        if s > M:
            break
    if (i+1) == len(arr) and s <= M:
        print(counts)
        ans = 1
    counts += 1

The inner loop uses break; there's no reason that the outer loop can't also, and that would simplify things slightly:

counts = 0
while True:
    s = 0
    for i in range(counts, len(arr)):
        s += int(arr[i]/(i+1-counts))
        if s > M:
            break
    if (i+1) == len(arr) and s <= M:
        print(counts)
        break
    counts += 1

When we get to the line

    if (i+1) == len(arr) and s <= M:

the first part of the condition can only fail because we called break in the inner loop, and in that case the second part of the condition also fails, so we can simplify that line to

    if s <= M:

Python has integer division: it's just that you need to use // instead of /. So

        s += int(arr[i]/(i+1-counts))

can be changed to

        s += arr[i] // (i+1-counts)

for readability.

It would be more Pythonic to operate on a slice rather than a range of indices. I.e. instead of

    for i in range(counts, len(arr)):
        s += arr[i] // (i+1-counts)

we would have

    for i, z_i in enumerate(arr[counts:]):
        s += z_i // (i+1)

The outer loop, on the other hand, could perfectly well be a range: it starts at 0 and is incremented once each time round the loop.

for counts in range(len(arr)+1):

Finally, there's the question of names. arr isn't great, but in context it passes. M comes from the problem statement. counts is, in my opinion, not at all helpful. I would call it offset, start, or something similar. And s is really partial_sum or weighted_sum.

This gives us tidied code:

for offset in range(len(arr)+1):
    partial_sum = 0
    for i, z_i in enumerate(arr[offset:]):
        partial_sum += z_i // (i+1)
        if partial_sum > M:
            break
    if partial_sum <= M:
        print(offset)
        break

As for speed: flooring discards information, so it's not easy to calculate the partial sum for a given offset any faster by using information from other offsets. Moreover, you can't use binary chop or some similar search because the partial sum is not monotonic in the offset: consider input array

[1000, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1000]

So I don't think there's much you can do to speed this up.


Compressing a stream asynchronously, and returning a byte[] Assuming I have to leave the contract intact (take byte[] and return byte[]), what's the correct way to structure this asynchronous method?

Will this deadlock running in ASP.NET in a web server thread?

public async Task<byte[]> CompressAsync(byte[] buffer)
{
    using (var outStream = new MemoryStream())
    {
        using (var tinyStream = new GZipStream(outStream, CompressionMode.Compress))
        using (var stream = new MemoryStream(buffer))
        {
            await stream.CopyToAsync(tinyStream);
        }

        return outStream.ToArray();
    }
}

I don't see any reason for the inner stream. Instead, you can just WriteAsync() the whole buffer directly into the (confusingly named) tinyStream.

Your code won't deadlock if used properly (i.e. asynchronously), but it will deadlock if someone synchronously waits for it (e.g. byte[] resultBuffer = CompressAsync(inputBuffer).Result;). To make sure this doesn't happen, add .ConfigureAwait(false) after every await.

You should do this for all your "library" code, that is code that doesn't need to run in the ASP.NET request context/GUI thread.

This method doesn't use any instance state. Why isn't it static?


C# Event handler for WPF MVVM So I've been learning about MVVM and I had a problem. I often had to start a thread to download or do some longer task in background and when it is done change some properties so the UI updates. The problem is that the UI will update only if I changed it from the main thread. So I often had to create DispatchTimers to wait for the thread to finish and then execute some code in main. Well this is my solution to it.
With this class you can simply go to your ViewModel constructor and do something like this:
ViewModelEventHandler.RegisterEventListener("EventName", EventAction);

EventAction being an action that is executed when the event is raised.
You can simply create an async Task that does its work in the background and it raises the event "EventName" and then EventAction() is executed in main thread. You just have to put new ViewModelEventHandler(); inside your main function.
I also use it a lot for communication between view models. You can raise an event in view model A and a completely unrelated view model B can execute code.
Here is my code:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Input;
using System.Windows.Threading;

namespace MVVM {
  public class EventData < T1, T2 > {
    public T1 eventState {
      get;
      set;
    }
    public T2 subs {
      get;
      set;
    }
  }

  class ViewModelEventHandler {
    // holds all events and their subscribed actions, the string is the event name, bool keeps track of if the event is currently raised, and List<Action> holds all the functions that will be executed when the event is raised
    private static Dictionary < string, EventData < bool, List<Action> >> eventList = new Dictionary < string, EventData <bool,List<Action> >> ();

    public ViewModelEventHandler() {
      DispatcherTimer timer = new DispatcherTimer();
      timer.Interval = TimeSpan.FromMilliseconds(1);
      timer.Tick += eventListen_Tick;
      timer.Start();
    }

    public static void RaiseEvent(string eventName) {
      try {
        eventList[eventName].eventState = true;
      }
      catch {}
    }

    public static void RegisterEventListener(string eventName, Action eventMethod) {
      foreach(var regEvent in eventList) {
        if (regEvent.Key == eventName) {
          eventList[eventName].subs.Add(eventMethod);
          return;
        }
      }

      eventList[eventName] = new EventData < bool, List < Action >> ();
      eventList[eventName].eventState = false;
      eventList[eventName].subs = new List < Action > ();
      eventList[eventName].subs.Add(eventMethod);

    }

    public static void RemoveEventListener(string eventName, Action eventMethod) {
      try {
        eventList[eventName].subs.Remove(eventMethod);
      }
      catch {}
    }

    public static void RemoveEvent(string eventName) {
      try {
        eventList.Remove(eventName);
      }
      catch {}
    }

    private void eventListen_Tick(object sender, EventArgs e) {
      foreach(var ev in eventList) {
        if (ev.Value.eventState) {
          ev.Value.eventState = false;
          foreach(Action eventSub in ev.Value.subs) {
            eventSub();
          }
        }
      }
    }

  }
}

Your approach is wrong. It creates too complex code for a very simple and common problem. To use a sort of event aggregator to report progress is overkill.
First notice that you generally distinguish between IO bound tasks and CPU bound tasks. CPU bound tasks are executed on a background thread. The performance of CPU bound tasks can be improved by using multi threading. For example calculations are CPU bound.
CPU bound tasks should be executed in parallel using background threads.
Reading from a file or download data are IO bound tasks. The performance of IO bound tasks cannot be improved using multiple threads, but by improving the IO devices. For example a download's performance is not limited by available CPU resources, but transfer rate or other network conditions and hardware. Using a thread for IO bound tasks is a waste of resources, since IO bound operations generally require almost no CPU resources.
IO bound tasks should be executed asynchronously or event-driven.
This means you should check the library you are using to download resources for asynchronous methods and await them e.g. await DownloadAsync(). Alternatively subscribe to corresponding API events e.g. DownloadProgressChanged. Every modern UI related API exposes asynchronous methods.
Since all DispatcherObject objects are associated with a Dispatcher and each thread has its own Dispatcher, you can't use or access a DispatcherObject on a different thread then the thread the DispatcherObject was created on. Except the DispatcherObject derives from Freezable and is in a frozen state.
To access a DispatcherObject from a different thread you must use the associated Dispatcher.
The following examples show how to update e.g., a Progressbar from a background thread:
MainWindow.xaml
<Window>
  <Progressbar x:Name="CompressionProgressbar" />
</Window>

MainWindow.xaml.cs
private async Task CompressData()
{
  // Start a background thread
  await Task.Run(() =>
  {
    // Access the DispatcherObject from a background thread
    // using the associated Dispatcher
    this.Dispatcher.InvokeAsync(() => this.CompressionProgressbar.Value = 33.33);
  });
}

Since .NET 4.5 the recommended approach is to use IProgress<T> to report progress (or delegate data back to the UI thread). The library offers a ready to use implementation, the Progress<T> class. The constructor captures the current SynchronizationContext to post the registered callback to it. Therefore it is important to create the instance on the UI thread and then pass it to the background thread.
private async Task CompressData()
{
  // Create an instance of Progress<T> using the constructor
  // to register the callback which will be executed on the current thread e.g. to report the progress
  var progressReporter = new Progress<double>(progressValue => this.CompressionProgressbar.Value = progressValue);

  // Start a background thread
  await Task.Run(() =>
  {
    // Access the DispatcherObject from a background thread
    // using the IProgress<T> instance
    progressReporter.Report(33.33);
  });
}


Duplication Elimination Project I am currently doing a project for school, and I am completely stumped. We are only on Chapter 8 of C# Development 1 and I have a question about the code.
The project is:

Use a one-dimensional array to solve the following problem:
Write an app that the user inputs numbers, each of which is between 10 and 100, inclusive. As each number is read, display it only if it’s not a duplicate of a number already read. Provide for the “worstcase,” in which all numbers are different. Use the smallest possible array to solve this problem. Display the complete set of unique values input after the user inputs each new value.

So far, this is what I have for the code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace MidtermHandsOn
{
class Program
{
static void Main(string[] args)
{
    //Create array from user input.
    Console.WriteLine("How many numbers will you enter?");
    int Array_Size = Convert.ToInt32(Console.ReadLine());
    int[] Array_Numbers = new int[Array_Size];
    //End array creation.

    //Input numbers into the array.
    int OuterCounter = 0;
    while (OuterCounter < Array_Size)
    {
        Console.Write("Enter number: ");
        int UserNumber = Convert.ToInt32(Console.ReadLine());

        if (UserNumber >= 10 && UserNumber <= 100) //Check if the numbers are between 10 and 100.
        {
            Array_Numbers[OuterCounter] = UserNumber;

            for (int InnerCounter = 0; InnerCounter < Array_Numbers.Length; InnerCounter++)
            {
                if (Array_Numbers[InnerCounter] > 0)
                {
                   Console.WriteLine(Array_Numbers[InnerCounter]);
                }
            }

            OuterCounter++;
        }
        else
        {
            Console.WriteLine("Number must be between 10 and 100");
        }
    }

    Console.ReadKey();
}
}
}

I can get the numbers to input and display in order, which is all well and good, but I cannot for the life of me get the duplicates eliminated.
Now, I'm not wanting the answer to the issue or anything, just wanting to get pointed in the right direction.
Is there ANY way to eliminate duplicates WITHOUT using Linq or Hashset or anything of the sort?

Not sure whether there's some trickery here... But a simple solution could be:

Simplest way would be an array of boolean for all possible numbers. When you enter 50, check position 50 and if it's false, it was the first time, now set that position to true so it can't be true the next time it is checked. Make it 90 in size and always add 10 to the index :).

bool[] hasBeenEntered = new bool[100-10];

while (true)
{
    int nr = Int.Parse(Console.Readline());
    if (nr >= 10 && <= 100)
    {
        if (hasBeenEntered[nr+10])
            continue;

        hasBeenEntered[nr+10] = true;
        Console.WriteLine(nr);
    }
}

There might be creative solutions using shorts and bit shifting to use a smaller array or actually use all the binary positions in an array of 2 Longs instead to make it a much smaller array. That's probably not the solution direction, but it would work pretty well...

bool[] hasBeenEntered = new long[2];

then check using bitmasks:

if (hasBeenEntered[nr/64] & (nr % 64) > 0)
    continue;

hasBeenEntered[nr/64] |= (nr % 64);
Console.WriteLine(nr);


A custom Pandas dataframe to_string method Oftentimes I find myself converting pandas.DataFrame objects to lists of formatted row strings, so I can print the rows into, e.g. a tkinter.Listbox. To do this, I have been utilizing pandas.DataFrame.to_string. There is a lot of nice functionality built into the method, but when the number of dataframe rows/columns gets relatively large, to_string starts to tank.

Below I implement a custom pandas.DataFrame class with a few added methods for returning formatted row lines. I am looking to improve upon the get_lines_fast_struct method.

import pandas

class DataFrame2(pandas.DataFrame):
    def __init__( self, *args, **kwargs ):
        pandas.DataFrame.__init__(self, *args, **kwargs)

    def get_lines_standard(self):
        """standard way to convert pandas dataframe
            to lines with fomrmatted column spacing"""
        lines = self.to_string(index=False).split('\n')
        return lines

    def get_lines_fast_unstruct(self):
        """ lighter version of pandas.DataFrame.to_string()
            with no special spacing format"""
        df_recs = self.to_records(index=False)
        col_titles = [' '.join(list(self))]
        col_data = map(lambda rec:' '.join( map(str,rec) ),
                         df_recs.tolist())
        lines = col_titles + col_data
        return lines

    def get_lines_fast_struct(self,col_space=1):
        """ lighter version of pandas.DataFrame.to_string()
            with special spacing format"""
        df_recs = self.to_records(index=False) # convert dataframe to array of records
        str_data = map(lambda rec: map(str,rec), df_recs ) # map each element to string
        self.space = map(lambda x:len(max(x,key=len))+col_space,  # returns the max string length in each column as a list
                         zip(*str_data))

        col_titles = [self._format_line(list(self))]
        col_data = [self._format_line(row) for row in str_data ]

        lines = col_titles + col_data
        return lines

    def _format_line(self, row_vals):
        """row_vals: list of strings.
           Adds variable amount of white space to each
           list entry and returns a single string"""
        line_val_gen = ( ('{0: >%d}'%self.space[i]).format(entry) for i,entry in enumerate(row_vals) )  # takes dataframe row entries and adds white spaces based on a format
        line = ''.join(line_val_gen)
        return line

Here I make some test data

import random
import numpy

#SOME TEST DATA
df = DataFrame2({'A':numpy.random.randint(0,1000,1000),
                 'B':numpy.random.random(1000),
                 'C':[random.choice(['EYE', '<3', 'PANDAS', '0.16'])
                      for _ in range(1000)]})

Method outputs

df.get_lines_standard()
#[u'   A         B       C',
# u' 504  0.924385      <3',
# u' 388  0.285854    0.16',
# u' 984  0.254156    0.16',
# u' 446  0.472621  PANDAS']
# ...

df.get_lines_fast_struct()
#['   A                 B      C',
# ' 504      0.9243853594     <3',
# ' 388    0.285854082778   0.16',
# ' 984    0.254155910401   0.16',
# ' 446    0.472621088021 PANDAS']
# ...

df.get_lines_fast_unstruct()
#['A B C',
# '504 0.9243853594 <3',
# '388 0.285854082778 0.16',
# '984 0.254155910401 0.16',
# '446 0.472621088021 PANDAS']
# ...

Timing results

In [262]: %timeit df.get_lines_standard()
10 loops, best of 3: 70.3 ms per loop

In [263]: %timeit df.get_lines_fast_struct()
100 loops, best of 3: 15.4 ms per loop

In [264]: %timeit df.get_lines_fast_unstruct()
100 loops, best of 3: 2.3 ms per loop

import pandas
np = pandas.np

What you are doing here is using the numpy that pandas imports, which can lead to confusion. There is an agreed standard to import pandas and numpy:

import pandas as pd
import numpy as np

And importing numpy yourself does not load the module twice, as imports are cached. Your import only costs a lookup in sys.modules because numpy already gets imported on the pandas import, but you add a lot of readability.

At the end you use random.choice() but you never imported random.

In get_lines_standard() you first convert the complete DataFrame to a string, then split it on the line breaks. In your example and then you slice the top 5 off it to display. The way your code works here, there is no way to only show the top 5 rows without rendering the complete DataFrame - which applies to all 3 methods.
Just to demonstrate the difference of slicing before and after (using random data generated at the end of your code but with 10k rows instead of 1k):

# both calls have the same output:

%timeit df.to_string(index=False).split('\n')[:5]
1 loops, best of 3: 1.51 s per loop

%timeit df[:5].to_string(index=False).split('\n')
100 loops, best of 3: 3.38 ms per loop

PS: I don't want to pep8ify you, but please don't line up your equal signs.

/edit:

Ok, let's focus on get_lines_fast_struct(). You are doing things manually for which there actually exist tools:

Creating a copy of a DataFrame with the same values as strings can be accomplished by str_df = self.astype(str)
The maximum lengths of cells per column of such a dataframe could be determined by self.spaces= [str_df[c].map(len).max() for c in str_df.columns]
For col_data you use a list comprehension that just call a method for each element, which is basically just map()
In _format_line() you fill up the strings with spaces on the left until they have length n+1 with n being the maximum col length by even mixing 2 styles of string formatting (old and new). string.rjust() does the same thing and might be faster.

All those things in mind the code might look like this:

def get_lines_fast_struct2(self, col_space=1):
    str_df = self.astype(str)
    self.space = [str_df[c].map(len).max() for c in str_df.columns]
    col_titles = map(_format_line2, [self.columns])
    col_data = map(_format_line2, str_df.to_records(index=False))
    return col_titles + col_data

def _format_line2(self, row_vals):
    return "".join(cell.rjust(width) for (cell, width) in zip(row_vals, self.space))

Let's compare this with the original in terms of speed and equality:

In [160]: %timeit df.get_lines_fast_struct()
100 loops, best of 3: 11.3 ms per loop

In [161]: %timeit df.get_lines_fast_struct2()
100 loops, best of 3: 9.78 ms per loop

In [162]: df.get_lines_fast_struct() == df.get_lines_fast_struct2()
Out[162]: True

Maybe there is even a better way with more pandas magic involved, but I am not that experienced with pandas yet.


Comparing two implementations of smoosh() In this question, the suggested solution for smoosh() is as shown below,

public static void smoosh(int[] a) {
    int originalPos = 0;
    int targetPos = 0;
    while (originalPos < a.length) {
        // Copy (and remember) one element to the correct position:
        int currentElement = a[targetPos++] = a[originalPos++];
        // Advance originalPos until a different element is found:
        while (originalPos < a.length && a[originalPos] == currentElement) {
            originalPos++;
        }
    }
    // Fill remaining elements:
    while (targetPos < a.length) {
        a[targetPos++] = -1;
    }
}

but, I felt that this well tested new solution shown below looks more readable than shown above,

  public static void smoosh(int[] a) {
    int saveOriginalPosition = 1;
    int targetPosition = 0, originalPosition = 0;
    while(targetPosition < a.length){
        if(saveOriginalPosition < a.length){
            for(originalPosition = saveOriginalPosition;originalPosition < a.length;originalPosition++)
                if(a[originalPosition] != a[targetPosition])
                {
                    a[++targetPosition] = a[originalPosition];
                    break;
                }
            saveOriginalPosition = originalPosition;
        }else{
            break;
        }
    }
    if(originalPosition == a.length){
        for(int setRemaining = targetPosition+1; setRemaining  < a.length; setRemaining++)
            a[setRemaining] = -1;
    }

 }

because point mentioned by 200_success, i.e., "However, being able to recognize the structure of the loop just by looking at the loop header gives it an advantage in readability." is considered in this new solution, whereas the inner while loop of the suggested solution does not look readable.

Please let me know your inputs on the changes.

I think there are issues with both implementaitons, though I do find the first more readable.

The main problem is that neither makes it immediately clear that we are assigning a new value to each element of the input array. I would expect an implementation to look something like this:

public static void smoosh(int[] input) {
  for (int i = 0; i < input.length; i++) {
    input[i] = ...
  }
}

This makes it clear to the reader that each element of the input array will be assigned to.

Now we can keep track of where the next element for our array is coming from:

public static void smoosh(int[] input) {
  int j = 0;
  for (int i = 0; i < input.length; i++) {
    input[i] = j < input.length ? input[j] : -1;
    j++;
    while (j < input.length && input[i] == input[j]) {
      j++;
    }
  }
}

It's getting hard to read again, so let's split the logic for j into its own method:

public static void smoosh(int[] input) {
  int j = 0;
  for (int i = 0; i < input.length; i++) {
    input[i] = j < input.length ? input[j] : -1;
    j = getNextDifferentIndex(input, j);
  }
}

/** Get the smallest integer j such that j > i and input[i] != input[j].
 * If no such element exists, return an integer greater than or equal to
 * input.length. */
private static int getNextDifferentIndex(int[] input, int i) {
  int j = i + 1;
  while (j < input.length && input[i] == input[j]) {
    j++;
  }
  return j;
}

Or if you want to stick to the 14-line limit mentioned in the previous question,

public static void smoosh(int[] input) {
  for (int i = 0, j = 0; i < input.length; i++, j = getNextDifferentIndex(input, j)) {
    input[i] = j < input.length ? input[j] : -1;
  }
}

private static int getNextDifferentIndex(int[] input, int i) {
  int j = i + 1;
  while (j < input.length && input[i] == input[j]) {
    j++;
  }
  return j;
}


Simple State Machine and Transition Table The goal is to have well defined state transitions, and the ability to provide the next event to execute.

I'd like to know if this is a proper implementation of State Machine, considering how states and transitionTable are defined, and how I handle event as input and output via nextEvent.

In many examples I cannot clearly define verbiage for state, so I defined them as verbs (notice ing suffix), as if it represents the ongoing progress of workflow. This may be wrong..

I also defined multiple events that could occur for a single state. For example, GetDeviceStatus and GetIntegrityStatus events both occur in RetrievingStatus state. This is also the case for Downloading state. You can see in the cases, when determining the next event, I need to check what the previous state was first.

If flawed, what are the pitfalls to my design, and how could it be improved? thanks.

enum ExampleState {
    case Initiate
    case Authorizing
    case RetrievingStatus
    case Downloading
    case Confirming
    case End
}

enum ExampleEvent {
    case InitiateSequence
    case GetAuthToken
    case GetDeviceStatus
    case GetIntegrityStatus
    case DownloadFromServer
    case DownloadToDevice
    case Confirm
}

class StateMachine {

var oldState: ExampleState!
var currentState: ExampleState!
var currentEvent: ExampleEvent!

var table: [ExampleState: [ExampleEvent: ExampleState]] = [.Initiate:         [.InitiateSequence:   .Authorizing],
                                                           .Authorizing:      [.GetAuthToken:       .RetrievingStatus],
                                                           .RetrievingStatus: [.GetDeviceStatus:    .Downloading, .GetIntegrityStatus: .Confirming],
                                                           .Downloading:      [.DownloadFromServer: .Downloading, .DownloadToDevice: .RetrievingStatus],
                                                           .Confirming:       [.Confirm:            .End]]

func nextEvent(event: ExampleEvent) -> EventExecutor? {

    let transitionState = table[currentState]![event]!
    let oldState = currentState

    switch (transitionState) {

        case .Initiate:
            currentState = .Initiate
            return InitiateSequence()

        case .Authorizing:
            currentState = .Authorizing
            return GetAuthToken()

        case .RetrievingStatus:
            currentState = .RetrievingStatus
            switch (oldState) {
            case .Authorizing: return GetDeviceStatus()
            case .Downloading: return GetIntegrityStatus()
            default: return nil
            }

        case .Downloading:
            currentState = .Downloading
            switch (oldState) {
            case .RetrievingStatus: return DownloadFromServer()
            case .Downloading: return DownloadToDevice()
            default: return nil
            }

        case .Confirming:
            currentState = .Confirming
            return Confirm()

        case .End:
            currentState = .End
            return nil
    }

}
}

Use case

@objc func rxExecute() {
    publisher
        .map { self.stateMachine.nextEvent(event: $0) } // $0 == event that just completed
        .flatMap { $0!.rxExecute() } // execute next event from state machine output
        .subscribe(subscriber) // handles errors/success from completed event
}

First there are a few things about design

Does you machine has something like 'error' that is not handled (e.g. by throwing an exception) or you have only 'outcomes' that handled in the same matter.

Do you expect branches in the FSM? Is it possible for an fsm method to fail? If possible how it is handled? E.g. ignored (say we rely on caller to retry on timer), raise failure event and queue it for execution, use return value as a guard to branch in state machine method.

Do you expect your code to maintained over 20+ years period?

After many attempts to make better existing FSMs, making my own, trying different frameworks and making my own frameworks too, I came to a trivially simple conclusion (aka 'dumb' FSM pattern): each event should be represented by a function and each function should contain a switch by state; or every state should be represented by a function and every function should contain a switch by event.

In the long run this is the cheapest solution. E.g. if you do not have branches/failure one can write a simple table driven fsm, however, once we got the first branch we need to either make it way more complicated or throw it away and rewrite using the 'dumb' approach


HackerRank Caesar Cipher, not using map, filter, reduce, regex Julius Caesar protected his confidential information by encrypting it in a cipher. Caesar's cipher rotated every letter in a string by a fixed number, K, making it unreadable by his enemies. Given a string,S , and a number, K , encrypt S.

Note: The cipher only encrypts letters; symbols, such as -, remain unencrypted.

func cipher(messageToCipher: String, k: Int)-> String{
  var  eMessage = ""
  let arr = messageToCipher.characters.map { String($0) }
  for ch in arr {
    for code in String(ch).utf8 {
        if (65<=code && code<=90) || (97<=code && code<=122) {
              if k > 26{
                    let value =  k % 26 == 0 ? k / 26 : k % 26
                    var pCode  = ((Int(code))  + (value))

                    if (pCode  > 122 && (97<=code && code<=122)) || (pCode  > 90 && (65<=code && code<=90)) {
                        pCode = pCode - 26
                    }

                   let s = String(UnicodeScalar(UInt8(pCode)))
                   eMessage = eMessage + s
                }else
                {
                    var pCode  = Int(code) + k

                    if (pCode  > 122 && (97<=code && code<=122)) || (pCode  > 90 && (65<=code && code<=90)) {
                        pCode = pCode - 26
                    }

                   let s = String(UnicodeScalar(UInt8(pCode)))
                   eMessage = eMessage + s
                }

        }
        else{
            eMessage = eMessage + ch
        }
    }
  }
  return eMessage
}

print(cipher(messageToCipher: "abc", k: 2) )

I solved this way and it was passed all test cases.

More information on this problem: Caesar Cipher

Enumerating UTF-8 characters: Your code uses two nested loops to encrypt letters
(based on the UTF-8 code):

var  eMessage = ""
let arr = messageToCipher.characters.map { String($0) }
for ch in arr {
    for code in String(ch).utf8 {
        if (65<=code && code<=90) || (97<=code && code<=122) {
            // ... translate `code` and append to `eMessage` ...
        }
        else{
            eMessage = eMessage + ch
        }
    }
}

This is unnecessary complicated and has an unwanted side effect if the message contains
non-ASCII characters (which are encoded as 2 or more UTF-8 code units):

print(cipher(messageToCipher: "ä €  ", k: 2) )
// ää €€€

This could be solved by adding a break in the else-case. The better solution is to
enumerate the UTF-8 code units directly:

var encoded: [UInt8] = []
for code in messageToCipher.utf8 {
    if (65<=code && code<=90) || (97<=code && code<=122) {
        // ... translate `code` and append to `encoded` ...
    }
    else{
        encoded.append(code)
    }
}
return String(bytes: encoded, encoding: .utf8)!

Code duplication: There is some duplicate code in

            if k > 26{
                let value =  k % 26 == 0 ? k / 26 : k % 26
                var pCode  = ((Int(code))  + (value))
                if (pCode  > 122 && (97<=code && code<=122)) || (pCode  > 90 && (65<=code && code<=90)) {
                    pCode = pCode - 26
                }
                let s = String(UnicodeScalar(UInt8(pCode)))
                eMessage = eMessage + s
            }else
            {
                var pCode  = Int(code) + k
                if (pCode  > 122 && (97<=code && code<=122)) || (pCode  > 90 && (65<=code && code<=90)) {
                    pCode = pCode - 26
                }
                let s = String(UnicodeScalar(UInt8(pCode)))
                eMessage = eMessage + s
            }

which is not necessary. The if-part works for the case k <= 26 as well.

Exceptional key values: The special handling of the case k % 26 == 0 causes
unexpected results:

print(cipher(messageToCipher: "abc xyz ABC XYZ", k: 26) ) // abc xyz ABC XYZ
print(cipher(messageToCipher: "abc xyz ABC XYZ", k: 52) ) // cde zab CDE ZAB

and is not needed. Negative key values are not handled at all:

print(cipher(messageToCipher: "abc xyz ABC XYZ", k: -2) )   // _`a vwx ?@A VWX
print(cipher(messageToCipher: "abc xyz ABC XYZ", k: -200) )
// Fatal error: Negative value is not representable

Both issues can be solved by computing the remainder of k modulo 26, as a number
in the range 0...25:

var value = k % 26
if value < 0 { value += 26 }

In addition, this computation can be done once, before entering the loop.

Other issues:

I would use some different argument/variable names, e.g. message instead of
messageToCipher (it is clear from the context and the function name that this string
should be encrypted), or  key instead of k.
There are unnecessary parentheses, e.g. in

var pCode  = ((Int(code))  + (value))

The usage of white space is not consistent, e.g. in

if (pCode  > 122 && (97<=code && code<=122)) || (pCode  > 90 && (65<=code && code<=90)) {

Summarizing the suggestions so far, we have

func cipher(message: String, key: Int) -> String {

    var offset = key % 26
    if offset < 0 { offset += 26 }

    var encoded: [UInt8] = []
    for code in message.utf8 {
        if (65 <= code && code <= 90) || (97 <= code && code <= 122) {
                var pCode  = Int(code) + offset
                if (pCode > 122 && (97 <= code && code <= 122)) || (pCode > 90 && (65 <= code && code <= 90)) {
                    pCode -= 26
                }
                encoded.append(UInt8(pCode))
        } else {
            encoded.append(code)
        }
    }
    return String(bytes: encoded, encoding: .utf8)!
}

Make it functional: If the encryption of a single character is moved to a separate
function then one can apply this to the UTF-8 view with map():

func cipher(message: String, key: Int) -> String {

    var offset = key % 26
    if offset < 0 { offset += 26 }

    func utf8encrypt(code: UInt8) -> UInt8 {
        // ... compute and return encrypted character ...
    }

    return String(bytes: message.utf8.map(utf8encrypt), encoding: .utf8)!
}

Further suggestions:

Use a switch-statement with range patterns instead of the
if-conditions:

switch code {
case 65...90:
    // ...
case 97...122:
    // ...
default:
    // ...
}

Use constants instead of the literal numbers, e.g.

let letter_A = 65
let letter_Z = 90
let letter_a = 97
let letter_z = 122

so that a future reader of your code does not have to guess what the numbers stand for.


Method to get next available filename I needed a utility function/method that would get the next available filename to save as.

An example would be, if I need to save a file as MyTestFile.html but it already exists.

This method would then check to see what the next availble filename would be, and it would return
MyTestFile2.html. If MyTestFile2.html exists, then it would return MyTestFile3.html, and so on.

I wrote the method, but it feels sloppy, and isn't really very elegant.

Any ideas for improving the readability of this code, or a better way to do it?

public string GetNextAvailableName(List<string> files, string baseFile)
{
    var baseFileWithoutExt = Path.GetFileNameWithoutExtension(baseFile);
    var baseExt = Path.GetExtension(baseFile);

    //clean files and get the ones containing oure baseFileName
    var cleanFiles = files
                        .Select(i => Path.GetFileNameWithoutExtension(i.ToLower()))
                        .Where(i => i.Contains(baseFileWithoutExt.ToLower()));

    var count = cleanFiles.Count();

    if (count == 0)
        return baseFile;

    int indexCount = 1;

    do
    {
        indexCount++;
    } while (cleanFiles.Contains(baseFileWithoutExt.ToLower() + indexCount));

    return baseFileWithoutExt + indexCount + baseExt;
}

This would then be called like this:

GetNextAvailableName(Directory.GetFiles("C:\\MyFiles\\", "MyTestFile.html");

Then if MyTestFile.html is already in use, it will return MyTestFileX.html, where X is the lowest positive integer where the filename doesn't exist yet.

I don't understand why you want to pass a list of files. Simply check the existence of your filename candidates. Start with the base filename. If it does not exist, start counting. In my eyes this a straightforward approach and therefore easy to understand.

public string GetNextAvailableFilename(string filename)
{
    if (!System.IO.File.Exists(filename)) return filename;

    string alternateFilename;
    int fileNameIndex = 1;
    do
    {
        fileNameIndex += 1;
        alternateFilename = CreateNumberedFilename(filename, fileNameIndex);
    } while (System.IO.File.Exists(alternateFilename));

    return alternateFilename;
}

private string CreateNumberedFilename(string filename, int number)
{
    string plainName = System.IO.Path.GetFileNameWithoutExtension(filename);
    string extension = System.IO.Path.GetExtension(filename);
    return string.Format("{0}{1}{2}", plainName, number, extension);
}


Typescript : Adding types to a function that takes in a string and returns a different promise based on the string I have a function that takes in a string and based on the string hits different APIs to fetch images.
I have solved it by writing an interface and then as the return type of the function, it returns a Promise<Interface> as shown in the code below.
    interface IAnimal {
      image?: string | null
      message?: string
      status: string
    }
    const getAnimal = async (inAnimal: string): Promise<IAnimal> => {
      const isDogWord: boolean = ["dog", "Dog", "DOG"].includes(inAnimal.trim())
      const isFoxWord: boolean = ["fox", "Fox", "FOX"].includes(inAnimal.trim())
      let result: IAnimal = { image: null, status: "tofetch" };
      if(isDogWord) {
        result = await fetch("https://dog.ceo/api/breeds/image/random").then((data) => data.json());
        if(result.status === "success") {
          return {
            image: result.message,
            status: result.status
          }
        }
      }
      if(isFoxWord) {
        result = await fetch("https://randomfox.ca/floof/").then((data) => data.json());
        return {
          image: result.image,
          status: "success"
        }
      }
      return {
        image: null,
        status: "failure"
      }
    };

This solves the typescript errors but I am not happy with making the image and message optional in the Interface. I am receiving { "message": "image__link", "status": "success or failure" } from dog API and {"image": "image__link", "status": "success or failure"} from the fox API. Is what I have done the right approach or do we have a better solution.

The problem here is that the interface IAnimal is not what those apis return. You want to return that interface from your function, but those APIs should have their own interface representing the response data.
interface DogApiResponse {
  message: string
  status: string
}

interface FoxApiResponse {
  image: string
}

interface Animal {
  status: string
  image: string | null
}

To generalize the set of available animal types I propose to also add interface for the fetching part of individual APIs.
declare type RandomAnimalFactory = () => Promise<Animal>

Now we declare two factories - for dogs and foxes
const dogFactory: RandomAnimalFactory = async () => {
  const result: DogApiResponse = await fetch(dogApiUrl).then(data => data.json())
  if (result.status !== 'success') {
    throw new Error('Dog API railure')
  }
  return {image: result.message, status: 'success'}
}

const foxFactory: RandomAnimalFactory = async () => {
  const result: FoxApiResponse = await fetch(foxApiUrl).then(data => data.json())
  return {image: result.image, status: 'success'}
}

The we can have the named animal factory work like this
declare type NamedAnimalFactory = (name: string) => Promise<Animal>

const createRandomNamedAnimalFactory = (randomFactories: {[key: string]: RandomAnimalFactory}): NamedAnimalFactory => {
  return async (name) => {
    const factory = randomFactories[name.trim().toLowerCase()]
    if (!factory) {
      return {image: null, status: 'failure'}
    }

    try {
      return await factory()
    } catch {
      return {image: null, status: 'failure'}
    }
  }
}

const getAnimal: NamedAnimalFactory = createRandomNamedAnimalFactory({
  dog: dogFactory,
  fox: foxFactory,
})
```


Sort pandas DataFrame first by subject ID then pick their corresponding values from other columns NOTES=[]
hadm=[]
for i in sorted(list(notes_df.SUBJECT_ID.unique())):
    #print(i)
    NOTES.append(list(notes_df[notes_df.SUBJECT_ID == i].TEXT))
    hadm.append(list(notes_df[notes_df.SUBJECT_ID == i].HADM_ID.astype(float)))
notes_1_df = pd.DataFrame(list(zip(sorted(list(notes_df.SUBJECT_ID.unique())),hadm,[item for sublist in NOTES for item in sublist])), columns =['SUBJECT_ID','HADM_ID','NOTES'])
notes_1_df.HADM_ID=notes_1_df.HADM_ID.astype(str)

The above code works fine to generate a new Dataframe with sorted values of subject ID and its corresponding values of other column. I used lists to store values and then used them to create a Dataframe.

But it is slow. How to make it fast?

notes_df structure:

ROW_ID|SUBJECT_ID|HADM_ID|TEXT
1      4           89     Here is the text
2      23          433    Here is the text,and so on
3      65          1212   Here is the text,and
4      914         2212   Here is the text,and so on
5      23          112    Here is the text,and so on
6      23          773    Here is the text,and so on
7      65          1210   Here is the text,and so
8      23          1212   Here is the text,and so on

What makes your code slow is the repeated calls to unique, and manually shunting all entries into lists repeatedly. Ideally you want to do it all in pandas for the most speed.

If I understand it correctly, you want to group by the subject ID, collect all hadm IDs and texts. In that case, you can just use pandas.DataFrame.groupby and pandas.DataFrame.aggregate to achieve (almost) the same result:

notes_1_df = notes_df.drop(columns=["ROW_ID"]) \
                     .groupby("SUBJECT_ID") \
                     .aggregate(list) \
                     .reset_index()

Which directly produces this output:

   SUBJECT_ID                HADM_ID                                               TEXT
0           4                   [89]                                 [Here is the text]
1          23  [433, 112, 773, 1212]  [Here is the text,and so on, Here is the text,...
2          65           [1212, 1210]    [Here is the text,and, Here is the text,and so]
3         914                 [2212]                       [Here is the text,and so on]

Whereas your code produces:

   SUBJECT_ID                        HADM_ID                       NOTES
0           4                         [89.0]            Here is the text
1          23  [433.0, 112.0, 773.0, 1212.0]  Here is the text,and so on
2          65               [1212.0, 1210.0]  Here is the text,and so on
3         914                       [2212.0]  Here is the text,and so on

This differs mainly in the notes. Here your code is a bit weird. Instead of doing what you did, let's just pick the first text from each subject for now.

I am assuming that having integers for the hadm ID is fine, otherwise you can be more specific and supply two different functions to aggregate, in which case you don't even need the drop anymore:

notes_1_df = notes_df.groupby("SUBJECT_ID") \
                     .aggregate({"HADM_ID": lambda x: list(map(float, x)),
                                 "TEXT": "first"}) \
                     .rename(columns={"TEXT": "NOTES"}) \
                     .reset_index()

   SUBJECT_ID                        HADM_ID                       NOTES
0           4                         [89.0]            Here is the text
1          23  [433.0, 112.0, 773.0, 1212.0]  Here is the text,and so on
2          65               [1212.0, 1210.0]        Here is the text,and
3         914                       [2212.0]  Here is the text,and so on

To see why I think your code produces weird results, let's replace the text with unique texts:

df["TEXT"] = "Text from subject " + df.SUBJECT_ID.astype(str) + ", hadm " + df.HADM_ID.astype(str)

Then my (last) code produces:

   SUBJECT_ID                        HADM_ID                             NOTES
0           4                         [89.0]      Text from subject 4, hadm 89
1          23  [433.0, 112.0, 773.0, 1212.0]    Text from subject 23, hadm 433
2          65               [1212.0, 1210.0]   Text from subject 65, hadm 1212
3         914                       [2212.0]  Text from subject 914, hadm 2212

Where each text is the first text from that actual subject. In contrast, your code produces:

   SUBJECT_ID                        HADM_ID                           NOTES
0           4                         [89.0]    Text from subject 4, hadm 89
1          23  [433.0, 112.0, 773.0, 1212.0]  Text from subject 23, hadm 433
2          65               [1212.0, 1210.0]  Text from subject 23, hadm 112
3         914                       [2212.0]  Text from subject 23, hadm 773

Note how the texts do not correspond to the same subjects anymore!


Maximum path sum of triangle of numbers I have implemented the maximum path sum of a triangle of integers (problem 18 in project euler) and I am wondering how I can improve my solution. My solution is below:

def problem18(triangle, height):
    return dp_triangle(triangle, height, 0, 0, dict())

def dp_triangle(triangle, height, row, col, maxs):
    if row == height - 1:
        return int(triangle[row][col])

    keyLeft = str(row + 1) + "," + str(col)
    keyRight = str(row + 1) + "," + str(col + 1)

    if keyLeft in maxs:
        maxLeft = maxs[keyLeft]
    else:
        maxLeft = dp_triangle(triangle, height, row + 1, col, maxs)
        maxs[keyLeft] = maxLeft

    if keyRight in maxs:
        maxRight = maxs[keyRight]
    else:
        maxRight = dp_triangle(triangle, height, row + 1, col + 1, maxs)
        maxs[keyRight] = maxRight

    return max(maxLeft, maxRight) + int(triangle[row][col])

To call this function I have the following:

def main(argv):
    with open(argv[1], 'r') as f:
        triangle = f.readlines()

    height = len(triangle)

    triangle = [x.strip().split() for x in triangle]

    print(problem18(triangle, height))

I just pass a text file with the triangle of numbers to the program. These numbers are separated by whitespace.

dp_triangle(triangle, height, row, col) depends on three things: dp_triangle(triangle, height, row + 1, col), dp_triangle(triangle, height, row + 1, col + 1), and triangle[row][col]. Therefore it's not necessary to cache more than one row at a time. With that insight you should be able to refactor it to not need dict() at all.

I actually prefer to solve this problem by going down rather than up. That is to say, I define my intermediate result as v(row, col) = max(v(row-1, col-1), v(row-1, col)) + triangle[row][col]. Then the double-array of triangle can be processed in order using a simple for loop, as opposed to in reverse using a slightly more complicated for loop.


Bash script to create directories and files with a numbered prefix I wrote a pretty simple bash script. It takes a directory with subdirectories with incremental prefixes in their names (01test, 02test, 03test), creates a new directory with the next highest prefix and creates a couple of files and fills them with some default text.

I'd like to do this in fewer lines of code and eliminate redundancies.

#!/bin/bash

LAST=`exec ls example_dir | sed 's/\([0-9]\+\).*/\1/g' | sort -n | tail -1`
PREFIX="${LAST:0:2}"
PREFIX=$((PREFIX + 1))
PREFIX="$PREFIX"_
ARGS=$@
TESTNAME="${ARGS// /_}"
DIRNAME="example_dir/$PREFIX$TESTNAME"

mkdir $DIRNAME
mkdir $DIRNAME/some_directory
mkdir $DIRNAME/expected

touch $DIRNAME/first.txt
touch $DIRNAME/second.txt
touch $DIRNAME/third.json

echo "{" >> $DIRNAME/test.json
echo -e "\t\"enabled\": true" >> $DIRNAME/test.json
echo "}" >> $DIRNAME/test.json

echo "{" >> $DIRNAME/some_directory/$PREFIX.json
echo -e "  \"entities\": [" >> $DIRNAME/some_directory/$PREFIX.json
echo "  ]" >> $DIRNAME/some_directory/$PREFIX.json
echo "}" >> $DIRNAME/some_directory/$PREFIX.json

tail must to read the whole file. Calling sort with a -r (aka --reverse option) you can use head instead.
touch takes multiple arguments. You may

touch $DIRNAME/first.txt $DIRNAME/second.txt $DIRNAME/third.json

and further use brace expansion:

touch $DIRNAME/{first.txt,second.txt,third.json}

I would consider here-documenting the json files instead of echoing them, along the lines of:

cat << EOF >> $DIRNAME/test.json
{
     "enabled": true
}
EOF


Nonlinear Scaling Normalization I have a procedure for normalizing variables, the details of which can be viewed in this white paper.

I would like to be able to make the routine scalable to handle any number of variables.  Below is the R code for a 4 variable example with a regular correlation coefficient substituted for the preferred nonlinear coefficient (for simplicity and demonstration purposes).  Any insights or comments are appreciated.

VN_Normalization <- function(A1, A2, A3, A4){

#Array1 Scaling Factor
RG_Factor_A1_A2<- mean(A1)/mean(A2)
RG_Factor_A1_A3<- mean(A1)/mean(A3)
RG_Factor_A1_A4<- mean(A1)/mean(A4)

#Array2 Scaling Factor
RG_Factor_A2_A1<- mean(A2)/mean(A1)
RG_Factor_A2_A3<- mean(A2)/mean(A3)
RG_Factor_A2_A4<- mean(A2)/mean(A4)

#Array3 Scaling Factor
RG_Factor_A3_A1<- mean(A3)/mean(A1)
RG_Factor_A3_A2<- mean(A3)/mean(A2)
RG_Factor_A3_A4<- mean(A3)/mean(A4)

#Array4 Scaling Factor
RG_Factor_A4_A1<- mean(A4)/mean(A1)
RG_Factor_A4_A2<- mean(A4)/mean(A2)
RG_Factor_A4_A3<- mean(A4)/mean(A3)

#A1 as Reference Gene
A1_1 <- A1
A2_1 <- A2*RG_Factor_A1_A2*abs((cor(A1,A2)))
A3_1 <- A3*RG_Factor_A1_A3*abs((cor(A1,A3)))
A4_1 <- A4*RG_Factor_A1_A4*abs((cor(A1,A4)))

#A2 as Reference Gene
A1_2 <- A1*RG_Factor_A2_A1*abs((cor(A1,A2)))
A2_2 <- A2
A3_2 <- A3*RG_Factor_A2_A3*abs((cor(A2,A3)))
A4_2 <- A4*RG_Factor_A2_A4*abs((cor(A2,A4)))

#A3 as Reference Gene
A1_3 <- A1*RG_Factor_A3_A1*abs((cor(A1,A3)))
A2_3 <- A2*RG_Factor_A3_A2*abs((cor(A3,A2)))
A3_3 <- A3
A4_3 <- A4*RG_Factor_A3_A4*abs((cor(A3,A4)))

#A4 as Reference Gene
A1_4 <- A1*RG_Factor_A4_A1*abs((cor(A1,A4)))
A2_4 <- A2*RG_Factor_A4_A2*abs((cor(A4,A2)))
A3_4 <- A3*RG_Factor_A4_A3*abs((cor(A4,A3)))
A4_4 <- A4

A1_Normalized <- (A1_1+A1_2+A1_3+A1_4)/4
A2_Normalized <- (A2_1+A2_2+A2_3+A2_4)/4
A3_Normalized <- (A3_1+A3_2+A3_3+A3_4)/4
A4_Normalized <- (A4_1+A4_2+A4_3+A4_4)/4

p = sample(rainbow(10))
boxplot(list(A1,A2,A3,A4,A1_Normalized,A2_Normalized,A3_Normalized,A4_Normalized),
  las=2, names=c("Array1","Array2","Array3","Array4",
  "Array1_Normalized","Array2_Normalized","Array3_Normalized","Array4_Normalized"),
    col=c("white","white","white","white",p,p,p,p))

}

I feel the key to generalizing your code is to store your variables into a matrix. Then let vectorized functions (colMeans, cor, *, etc.) do their magic:

A <- cbind(A1, A2, A3, A4)

VN_Normalization <- function(A) {
   m  <- colMeans(A)
   RG <- m %o% (1/m)
   scales <- colMeans(RG * abs(cor(A)))
   A_Normalized <- t(t(A) * scales)

   n <- ncol(A)
   i <- seq_len(n)
   labels <- c(sprintf("Array%i", i),
               sprintf("Array%i_Normalized", i))
   boxplot(cbind(A, A_Normalized),
           las = 2, names = labels,
           col = c(rep("white", n), rainbow(n)))
}

Please let me know if I missed anything (I assumed A1, A2, etc. were numeric vectors of equal lengths.)


Multiple SELECT with LEFT JOIN to order results I have two tables, looking like this:

Table 1

id | name |          date
---+------+---------------------
 1 | zaak |  2016-08-03 11:37:40  -- top 1 dates, name #5
 2 | aap  |  2016-08-01 11:40:35
 3 | corn |  2016-08-02 11:42:16  -- top 3 dates, name #3
 4 | quar |  2016-08-02 11:42:28  -- top 2 dates, name #4
 5 | alfa |  2016-07-04 11:44:29

Table 2

id | open | name  | name_id | type
---+------+-------+---------+-----
 1 |   1  | aap   |    2    |   2  -- type = 2, open = 1 : 1
 2 |   1  | quar  |    4    |   2  -- type = 2, open = 1 : 1
 3 |   1  | quar  |    4    |   2  -- type = 2, open = 1 : 1
 4 |   1  | dinky |    1    |   1  -- type = 1, open = 1 : NULL
 5 |   2  | quar  |    1    |   1  -- type = 1, open = 2 : NULL

The result I need is id and name from table 1 and open from table 2.

The requirements:

from table 1: select the newest three by date
order those three by name from A - Z
from table 2: select open

IF name not there return NULL
IF type <> 2  return NULL
IF open <> 1  return NULL
ELSE              return 1

With a lot of trial and error I constructed this query:

-- get id, name and open only once
SELECT DISTINCT t1.id , t1.name , t3.open

-- select names from table 1, order them by date and return first 3
FROM (SELECT id , name FROM table1 ORDER BY date DESC LIMIT 3) AS t1

-- get open from table 2 IF id is there, open = 1 and type =2
LEFT JOIN ( ( SELECT open , name_id
              FROM table2
              WHERE open=1 AND type=2
            ) AS t3
          )
ON t1.id = t3.name_id

-- order by name from A-Z
ORDER BY t1.name ASC

The result I get is correct:

id | name | open
---+------+-----
 3 | corn | NULL  (correct)
 4 | quar | 1     (correct)
 1 | zaak | NULL  (correct)

As my knowledge of MYSQL is limited, I wonder

would this query always return the desired result? (the data in the tables is limited at the moment and I have to change values by hand to see different results, so testing all variations proves a bit difficult)
is there anything I could improve on this query, or should I construct an entirely different query? (as triple SELECT to get the desired result seems a bit overdone)

First of all you don't need so many subqueries. all of your code can be done in given way

SELECT DISTINCT t1.id, t1.name, t2.open
FROM table1 t1
LEFT JOIN table2 t2 ON t1.id = t2.name_id AND t2.open = 1 AND type = 2
ORDER BY t1.date DESC, t1.name ASC LIMIT 3

Works exactly the way you want it, plus is almost 3 times faster on my machine. (0.0012 seconds vs 0.0034)

for additional reading on subqueries and joins https://stackoverflow.com/questions/3856164/sql-joins-vs-sql-subqueries-performance
https://stackoverflow.com/questions/2577174/join-vs-sub-query

Second point is that I think you should not be storing name value in table2. i don't know your project requirements or specifics but one of the major point of relational databases is to diversify data and avoid copies of the same data in different places.

table2 is connected with table1 with name_id which means it indirectly provides name value and i would suggest removing that column from table2 if it's not absolutely neccessery for your project, or it would mean u should rewrite  big portion of your code.


Reporting progress using a Follower My goal is to implement a function that accepts a Follower object that will be used to report progress through a complex function (a depth-first traversal of a graph). Here's my attempt:

pub trait Follower<'a, N> {
    fn root(&mut self, node: &'a N);
}

pub fn depth_first<'a, N>(
    node: &'a N, follower: &mut dyn Follower<'a, N>
) {
    follower.root(node);

    // TODO
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::fmt::Display;

    struct DebugFollower<'a, N> {
        nodes: Vec<&'a N>
    }

    impl<'a, N: Display> DebugFollower<'a, N> {
        fn new () -> Self {
            DebugFollower { nodes: vec![ ] }
        }

        fn to_string(&self) -> String {
            self.nodes.iter().map(ToString::to_string).collect()
        }
    }

    impl<'a, N> Follower<'a, N> for DebugFollower<'a, N> {
        fn root(&mut self, item: &'a N) {
            self.nodes.push(item);
        }
    }

    #[test]
    fn walks_p1() {
        let mut follower = DebugFollower::new();

        depth_first(&0, &mut follower);

        assert_eq!(follower.to_string(), "0");
    }
}

The code compiles and performs as expected. However, I suspect that, in trying to silence the many compile errors I encountered, I may have added unnecessary lifetime parameters. Previous experience has shown me that it's very easy to get into a situation wherein so many useless parameters have been added that removing them one at a time isn't possible. Still, I have tried to do that and can't find any lifetime parameters that can be removed.

In particular, I'm suspicious of the lifetime parameter on Follower, which appears to leak implementation details from DebugFollower. For example, removing the nodes property from DebugFollower, and updating the rest of the code, Follower no longer needs a lifetime parameter.

Does this example follow idiomatic Rust use of lifetime parameters? If not, how can I improve the use of lifetime parameters here?

You can get rid of the lifetimes completely, just replace every instance of:

&'a N

with just:

N

See Playground

But why? Well, in your current code N is inferred to be i32. However, with my suggested changes N will be inferred to be &'a i32. If you allow Rust to infer the reference it will also infer the correct lifetime for you.

But does this, as you ask, leak implementation details of DebugFollower? Not from a Rust perspective. In Rust, ownership details are very much part of the interface and not just an implementation detail. The key issue is that the Follower trait has to specify how long the references will be valid so that DebugFollower knows how long it is allowed to store those references.


Should I use an intermediate variable in a complex C# expression? Can anyone please tell which one is the more effective way to write C# code, among the two options below?

Option A:

internal static string GetLast3Years()
{
    return (DateTime.Now.Year - 2).ToString() + "," + (DateTime.Now.Year - 1).ToString() + "," + DateTime.Now.Year.ToString();
}

Or option B:

internal static string GetLast3Years()
{
    int currentYear = DateTime.Now.Year;
    return (currentYear - 2).ToString() + "," + (currentYear - 1).ToString() + "," + currentYear.ToString();
}

Note that semantically these options are not equal.

There is a chance (very-very small) that first option will output smth like 2011,2012,2014, so second option is better at least because it will always generate consistent results (3 consecutive years).

But second option is also quite cluttered. There is a string.Join method that is able to combine different values with separator, so your code may look like:

internal static string GetLast3Years()
{
    int currentYear = DateTime.Now.Year;
    return string.Join(",", currentYear - 2, currentYear - 1, currentYear);
}

But the question is still why would you need a string representation of last 3 years, comma-separated. I would consider having a method that will return enumerable of last N years instead...


Simplifying this factorial function def factorial(t,n):
    if n == 1 : return t

    else: x = (t * (n-1))

    n = n-1

    return factorial(x,n)

print factorial(6,6)

I can't seem to figure out a way to just stick to requiring one parameter input while keeping the program small. I also want the function to remain recursive (trying to work on my recursive thinking).

If you have any other cooler simplifications or other fancy methods, please show it off.

First, if n == 1the method returns, so there is no need for the else.
Let us see what 6! would result in: 6 * 5! which is just 6 * 5 * 4! and so on.
Now we see a patter: n! = n * (n-1)!
Keeping this in mind we can refactor the given method to only take one parameter and to be recursive.
def factorial(n):
    if n < 0 :
        raise ValueError("Negative values are not allowed.")
    if n == 0 : return 1

    return n * factorial(n-1)

print factorial(6)

As James Snell stated correctly in his comment, we can replace the n == 1 check with a n <= 2 check if we assume n >= 1.
As jonrsharpe stated correctly in his comment, 0! should return 1 and also negative numbers don't have a factorial.
As Simon André Forsberg stated correctly in his comment, rasing an error for a negative parameter should be done.


Calculating a total price by using switch statement and sentinel controlled loop My homework is to make a program that calculates the total value of products sold.  It works, but I use the if statement because it was asking for the quantity on the output before ending the loop. And for the same reason, I couldn't make the program ask the user again to enter a number from 1 to 5.

import java.util.Scanner;

public class Sales {

    public static void main(String[] args) {

        Scanner input = new Scanner(System.in);
        double sum = 0;
        int n = -1 ;
        while(n !=0 ) {

            System.out.println("Enter product number 1-5 (Enter  to stop)");
            n = input.nextInt();

            if (n==0)
                break;

            System.out.println("Enter quantity of product");
            int q = input.nextInt();

            switch (n) {
                case 1:
                    sum += 2.98*q;
                    break;
                case 2:
                    sum += 4.50*q;
                    break;
                case 3:
                    sum += 9.98*q;
                    break;
                case 4:
                    sum += 4.49*q;
                    break;
                case 5:
                    sum += 6.87*q;
                    break;
                default:
                    sum += 0;
                    break;
            }
        } System.out.printf("Total cost is $%.2f",sum);
    }
}

Prefer not to use single characters as variable names (except for simple loop counters).  Saving a few keystrokes is not worth it if your readers cannot understand your code.

Based on the context, it looks like n and q should respectively be named productNumber and productQuantity.
The sum += 0 in the default looks unnecessary.  Just remove it if so, even if it's only there to have the default do something.  Keep things simple.


Buidling classes with user input and calling methods Learned alot in this first week, following PCC, but biggest leaps came from the community and others giving their time to help, thank you!

All input appreciated!

ObjectiveGoals were to fill a class with user inputs from multiple users and then be able to call methods to each user input created class. (... did I say that right O.o)

Code

class User():
    def __init__(a, first_name, last_name, city, age):
        a.first_name = first_name.title()
        a.last_name = last_name.title()
        a.city = city.title()
        a.age = age

    def describe_user(a):
        print("-----")
        print("First Name" + " : " + a.first_name)
        print("Last Name" + " : " + a.last_name)
        print("City" + " : " + a.city)
        print("Age" + " : " + a.age)

def ask_user(message=''):
    user_input = ''
    while not user_input:
        user_input = input(message)
    return user_input

def form_complete(values, placement, length):
    placement = []
    while len(placement) < length:
        first_name = ask_user("Enter First Name: ")
        last_name =  ask_user("Enter Last Name: ")
        city = ask_user("Enter City: ")
        age = ask_user("Enter Age: ")
        values = User(first_name, last_name, city, age)
        placement.append(values)
    return placement

if __name__ == '__main__':

    users = form_complete('user', 'users', 3)
    for a in range(len(users)):
        users[a].describe_user()

Output

xenial)vash@localhost:~/pcc/9$ python3 3.py
Enter First Name: vash
Enter Last Name: the stampede
Enter City: gunsmoke
Enter Age: 131
Enter First Name: spike
Enter Last Name: spiegel
Enter City: mars
Enter Age: 27
Enter First Name: holden
Enter Last Name: caulfield
Enter City: new york city
Enter Age: 16
-----
First Name : Vash
Last Name : The Stampede
City : Gunsmoke
Age : 131

Greetings Vash!
-----
First Name : Spike
Last Name : Spiegel
City : Mars
Age : 27

Greetings Spike!
-----
First Name : Holden
Last Name : Caulfield
City : New York City
Age : 16

Greetings Holden!
(xenial)vash@localhost:~/pcc/9$

Python has something called magic methods (sometimes also called dunder methods, but the other name is way cooler).

These methods have special names and enable custom classes to use built-in functionality. If, for example you write a custom numeric class, you would want to be able to do e.g. a + b, with at least one of them being an instance of your class (and the other one either as well or even just a plain number). In order to achieve this, you would just implement an appropriate __add__ method for your class.

Here is a good list going through all available magic methods.

In your class, we can use the __str__ method. It gets used when you call str(obj), print(obj), "{}".format(obj) (unless you also implement the __format__ method):

class User:
    def __init__(self, first_name, last_name, city, age):
        self.first_name = first_name
        self.last_name = last_name
        self.city = city
        self.age = age

    def __str__(self):
        s = ["-----",
             f"First Name : {self.first_name}",
             f"Last Name : {self.last_name}",
             f"City : {self.city}",
             f"Age : {self.age}"]
        return "\n".join(s)

Note that I made some more changes:

The empty parenthesis after the class name are not needed. You only need parenthesis if you want to inherit from some other class.
I also called the first argument of the methods self, which is the customary name for it in Python.
I did not build the string using simple addition, since that is quite slow in Python (strings are immutable objects, so each string addition involves creating a new string of the right length and copying the content of the two strings being added).
And finally, I used f-strings to make the formatting a bit easier.

In your form_complete function (which IMO should be called complete_form, since that describes an action), you could use a list comprehension:

def complete_form(n_users):
    return [User(ask_user("Enter First Name: ").title(),
                 ask_user("Enter Last Name: ").title(),
                 ask_user("Enter City: ").title(),
                 ask_user("Enter Age: ", type=int)
            for _ in range(n_users)]

Note that I moved the alls to str.title here, since it should be the responsibility of the user of your class to pass in correct values (for example, what if your user enters a name for which title() does not make sense? There are also names that start with lowercase letters. Etc...)

I also removed values and placement from the signature, since you override them right away anyways.

And finally, I added a keyword argument to ask_user, which allows you to validate that age is actually an integer:

def ask_user(message='', type=str):
    user_input = ''
    while not user_input:
        try:
            user_input = type(input(message).strip())
        except ValueError:
            continue
    return user_input

By default, this just calls the str function on whatever your user supplied. If you instead specify type=int, it will call int on the input, which raises a ValueError if it cannot parse the input it received as an int.

This has only one weakness right now: If the user wants to enter something that parses as 0, they are stuck in an infinite loop (so no users under the age of 1 allowed).

I also added a call to str.strip, so user names like  (space) and  (tab) are not allowed either.


Find the date for the nearest day of the week, in this case Fri & Sat General objective: Find the date (day of the month) of the nearest specified day of the week and output.

More specifically: Focusing on events which take place every Fri & Sat, want the page to automatically update dates to next events. HOWEVER for clarity the page should never display Fridays and Saturdays which are NOT part of the same weekend. This would only happen on a SAT. So, if today's date is SAT, the displayed date for FRI should be yesterday (the immediately previous Friday), while the displayed date for SAT should be today's date.

It seems to work now.

Does this code follow common best practices?

Can you foresee cases where it would fail?

I am quite new to JavaScript and coding as you can probably tell.

(function() {

  var startingDate = new Date();

    function dateOfNearestDay(startingDate, nearestDay) {

    var nearestTime = new Date(startingDate.getTime());

    if (startingDate.getDay() == 6 && nearestDay == 5) {

      nearestTime.setDate((startingDate.getDate() + (7 + nearestDay - startingDate.getDay()) % 7) - 7);
    }
    else {

      nearestTime.setDate(startingDate.getDate() + (7 + nearestDay - startingDate.getDay()) % 7);

    }

    var nearestMonth = nearestTime.getMonth();
    var nearestDate = nearestTime.getDate();
    var nearestBoth = [nearestMonth, nearestDate];

    return nearestBoth;
  }

  monthNames = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'];

  var nearestMonthFriday = monthNames[dateOfNearestDay(startingDate, 5)[0]];
  var nearestDateFriday = dateOfNearestDay(startingDate, 5)[1];
  var nearestMonthSaturday = monthNames[dateOfNearestDay(startingDate, 6)[0]];
  var nearestDateSaturday = dateOfNearestDay(startingDate, 6)[1];

  var fridaysDate = document.getElementById('fridaysDate');
  var saturdaysDate = document.getElementById('saturdaysDate');

  fridaysDate.textContent = nearestMonthFriday + " " + nearestDateFriday;
  saturdaysDate.textContent = nearestMonthSaturday + " " + nearestDateSaturday;

}());

I think dateOfNearestDay would be more intuitive and useful if it would return the date object,
after all you are dealing with dates here. This would also simplify the caller code, who now works with a date object instead of an array.

Keep a separated method to format the date the way you want to and you are good to go.
Besides of this two things you wrote nice code overall.

(function() {

  var startingDate = new Date();

  function dateOfNearestDay(startingDate, nearestDay) {
    var nearestTime = new Date(startingDate.getTime());

    if (startingDate.getDay() == 6 && nearestDay == 5) {
        nearestTime.setDate((startingDate.getDate() +
           (7 + nearestDay - startingDate.getDay()) % 7) - 7);
    }
    else {
        nearestTime.setDate(startingDate.getDate() +
           (7 + nearestDay - startingDate.getDay()) % 7);
    }

    return nearestTime;
  }

  function getMonthAndDay(date){
      var monthNames = ['January', 'February', 'March', 'April',
      'May', 'June', 'July', 'August', 'September',
      'October', 'November', 'December'];

      return monthNames[date.getMonth()]+ " " + date.getDate();
  }

  var nearestDateFriday = dateOfNearestDay(startingDate, 5);
  var nearestDateSaturday = dateOfNearestDay(startingDate, 6);

  var fridaysDate = document.getElementById('fridaysDate');
  var saturdaysDate = document.getElementById('saturdaysDate');

  fridaysDate.textContent = getMonthAndDay(nearestDateFriday );
  saturdaysDate.textContent = getMonthAndDay(nearestDateSaturday );

}());


PropertyInfo GetValue and Expression Cache GetValue I have implemented PropertyInfo GetValue and Expression Cache GetValue by Logic from Reflection vs. compiled expressions vs. delegates - Performance comparison,but I'm sure that there are better ways to do every thing that I did. I would appreciate any feedback on this code, which does work according to spec:

MyLogic

  Asynchronous nonblocking, which does not block thread when adding a Cache
  Dictionary Key use propertyInfo GetHash()
  if dictionary cache containskey then use compiler expression function else use Reflection GetValue

My Class

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq.Expressions;
using System.Reflection;
using System.Threading.Tasks;

public class TestService
{
    private static Dictionary<int, object> ExpressionCache = new Dictionary<int, object>();

    public IEnumerable<string> Execute<T>(IEnumerable<T> enums)
    {
        var expressionCache = (enums);
        var props = typeof(T).GetProperties();
        foreach (var e in enums)
        {
            foreach (var p in props)
            {
                var func = GetOrAddExpressionCache<T>(p);
                var value = string.Empty;
                if (func == null)
                {
                    Debug.WriteLine("Use Reflection");
                    value = p.GetValue(e).ToString();
                }
                else
                {
                    Debug.WriteLine("Use Expression");
                    value = func(e);
                }
                yield return value;
            }
        }
    }

    private Func<T, string> GetOrAddExpressionCache<T>(PropertyInfo prop)
    {
        var key = prop.GetHashCode();
        if (ExpressionCache.ContainsKey(key))
        {
            var func = ExpressionCache[key] as Func<T, string>;
            return func;
        }else{
            Task.Run(()=>AddExpressionCacheAsync<T>(prop));
            return null;
        }
    }

    //Asynchronous nonblocking, which does not block a thread when adding a Cache
    private async Task AddExpressionCacheAsync<T>(PropertyInfo propertyInfo)
    {
        var key = propertyInfo.GetHashCode();
        if (!ExpressionCache.ContainsKey(key))
        {
            var func = GetValueGetter<T>(propertyInfo);
            ExpressionCache.Add(key, func);
        }
        await Task.Yield();
    }

    private static Func<T, string> GetValueGetter<T>(PropertyInfo propertyInfo)
    {
        var instance = Expression.Parameter(propertyInfo.DeclaringType);
        var property = Expression.Property(instance, propertyInfo);
        var toString = Expression.Call(property, "ToString", Type.EmptyTypes);
        return Expression.Lambda<Func<T, string>>(toString, instance).Compile();
    }
}

Demo

class Program{
    public static void Main(string[] args){
        var data = new[] { new { Name = "Henry", Age = 25 } };
        for (int i = 0; i < 2; i++)
        {
            var service = new TestService();
            var result = service.Execute(data).ToArray();
        }
    }
}

/* Console :
        Use Reflection
        Use Reflection
        Use Expression
        Use Expression
*/

Problems

Both the reflection and expression-based approach fail to take null values into account (NullReferenceException).
They also fail to take types with indexers into account (TargetParameterCountException).
Hash-codes are not unique identifiers. Different properties (and objects in general) can have the same hash code - that just means that they might be equal. Use prop.MetadataToken in combination with the module they're declared in (prop.Module.ModuleVersionId).
Compiling expressions asynchronously makes this a lot more complex than it needs to be, and that complexity is not properly taken care of:

There's no guarantee that compilation will be finished when the next object is processed. This can result in multiple compilations for the same property. That's a waste of work.
The results of these compilations are added to a Dictionary, which is not thread-safe. In the best case, adding a key that already exists will throw an exception. In the worst case, the dictionary's internal state will become corrupted.
Those exceptions are not being caught. Before .NET Framework 4.5, that would've caused a crash, and in 4.5 and higher it may still do so, depending on certain settings.

Other improvements

Why does Execute take a sequence of items, instead of a single item? Flattening results is easy (data.SelectMany(service.Execute)), 'unflattening' is not - the caller would have to figure out the number of properties, and you'd have to write a method to split up a single sequence into sub-sequences.
Instead of doing ExpressionCache.ContainsKey(key), followed by ExpressionCache[key], use TryGetValue. This lets you check the presence of a key and get its associated value with a single lookup.
There's no point in making AddExpressionCacheAsync async. It's not doing any asynchronous work (that await Task.Yield() is useless). You're already calling this method from within a Task.Run call, so it'll be executed asynchronously anyway.
In GetValueGetter, use nameof(object.ToString) instead of "ToString".

Readability issues

Some names are not very descriptive: Execute -> GetPropertyValues, enums -> items, func -> compiledExpression, GetValueGetter -> CompileGetValueExpression.
There are several abbreviations that do not improve readability: e -> item, props -> properties, p -> property.


Stack Exchange Data Explorer query to find upvoted questions with only downvoted answers I have written a SQL query against the Stack Exchange Data Explorer that identifies answered questions with a positive score for which all the answers have a negative score (this was an answer to this Meta Stack Exchange post, though the code in that post is different from what I've posted below).

It's a fairly straightforward query; you can see it and run it on the Data Explorer (where you can also examine the schemata of the tables), and I have reproduced it below for your convenience.

SELECT
  q.Id AS [Post Link], -- this alias makes the Data Explorer do magic link rendering
  MAX(q.Score) AS [Score],
  COUNT(a.Id) AS [AnswerCount],
  AVG(CAST(a.Score AS float)) AS [AvgAnswerScore]
FROM
  Posts AS q
  -- INNER JOIN => At least one answer
  INNER JOIN Posts AS a
    ON q.Id = a.ParentId
WHERE
  -- Question has a positive score
  q.Score >= 1
GROUP BY
  q.Id
HAVING
  -- Zero answers with a non-negative score
  SUM((CASE WHEN a.Score >= 0 THEN 1 ELSE 0 END)) = 0
ORDER BY
  [Score] DESC,
  [AnswerCount] DESC

While I of course would love to hear anything you have to say about this code, I am particularly interested in the following:

Stylistic tips - coming from the perspective of a PEP8-adhering Python guy, the huge inhomogeneity in SQL coding styles makes me queasy.
MAX(q.Score) looks really stupid and is semantically wrong, since there's obviously only one q.Score for each q.Id since they come from the same table, and q.Id is the primary key of that table. But putting q.Score into the GROUP BY also seems weird to me, since that's not semantically what I'm doing either. Is there a better way to approach this?
How do the performance characteristics of the HAVING clause here differ from instead using a WHERE clause of the form below?

q.Score >= 1
AND NOT EXISTS (
  SELECT 1
  FROM Posts AS a2
  WHERE a2.ParentId = q.Id AND a2.Score >= 0
)

My sense is that HAVING SUM(...) = 0 is probably better, since, uh... there doesn't need to be a new subquery for each row of the outer query? But I don't know if that is true or even makes sense.

Answering your questions in reverse order…

Your HAVING clause is clearer than than an anti-join condition in the WHERE clause.  It is also more efficient — tick the "Include execution plan" checkbox to see how much more complicated the anti-join is.  You could do better, though:  MAX(a.Score) < 0 also means that all answers have a net negative score.

Given the choice between selecting MAX(q.Score) and grouping by q.Score, I'd prefer the latter.  Both are annoying, but the SQL standard requires that you do one or the other, so that all columns are either mentioned in GROUP BY or are aggregated.

I can provide some style tips, but keep in mind that these are my personal recommendations.  I don't believe that there is a standard SQL style guide — and certainly nothing as formalized as PEP 8.

Fill in the description to your Data Explorer query.  Otherwise, there is a huge gap between the information contained in the title and the SQL.  Detail exactly what the query does.  Mention your motivation for writing the query.  State what the sort order is.
Indent more.  The FROM, WHERE, GROUP BY, HAVING, and ORDER BY clauses are all subsidiary to the SELECT, and therefore should be at the next indentation level.
Finishing the query with a semicolon is a good habit.
Move comments to the right to be less obtrusive.
I like to put commas before the selected columns.  That convention makes it easier to add and remove columns.
Prefer comparing with 0 instead of comparing with any other constant.

SELECT q.Id AS [Post Link] -- Alias makes Data Explorer do magic link rendering
     , q.Score AS [Score]
     , COUNT(a.Id) AS [AnswerCount]
     , AVG(CAST(a.Score AS float)) AS [AvgAnswerScore]
    FROM Posts AS q
        INNER JOIN Posts AS a
            ON q.Id = a.ParentId
    WHERE
        q.Score > 0        -- Question has a positive score
    GROUP BY
        q.Id, q.Score
    HAVING
        MAX(a.Score) < 0   -- Best answer has a negative score
    ORDER BY
        [Score] DESC,
        [AnswerCount] DESC;

I'd like to point out: good job remembering to CAST(a.Score AS float) before taking the average.  SQL Server is tricky, in that AVG() of ints is an int.


Steam whitelist checker for CS 1.6 I wrote some code that reads the output of the rcon users command on the CS 1.6 console. It then notifies me of any players that are not in my whitelist.csv or are in my blacklist. The program requires 3 files as inputs:

A text file containing the raw output of the rcon users command on CS 1.6 console.
A CSV file containing the SteamID and name of the white-listed players.
A CSV file containing the SteamID and name of the black-listed players.

#!/usr/bin/env python3
import csv
import re

class WhiteListVerifier:
    """
    Class to check if non-white-listed players are playing CS.
    """
    def __init__(self, users_filepath, whitelist_filepath, blacklist_filepath):
        """
        Get filepaths and load the following files: users file , whitelist file and blacklist file

        users_filepath: Path to a text file which contains the raw output of the 'rcon users' in the in-game console.
        whitelist_filepath: Path to a csv file which contains Steam IDs and player-names of the white-listed players.
        blacklist_filepath: Path to a csv file which contains Steam IDs, player-names and alert-message for the black-listed players.
        """
        with open(users_filepath) as f:
            self.users_rawcontents = f.read()
        self.whitelist = list(csv.DictReader(open(whitelist_filepath)))
        self.blacklist = list(csv.DictReader(open(blacklist_filepath)))

    def get_playername(self, steamid, player_records):
        """
        Function to return name of a person if steamid exits in given records
        and return 'Unknown' otherwise.
        """
        for player in player_records:
            if steamid == player['Steam_ID']:
                return player['Player_Name']
        return 'Unknown'

    def get_active_users_steamids(self):
        """
        Function to extract the Steam IDs using output of 'rcon users' command.
        """
        active_users_steamids = re.findall(r'STEAM_[0-5]:[01]:\d+', self.users_rawcontents)
        return active_users_steamids

    def verify_steamids(self):
        """
        Function to alert if people other than the ones mentioned in the whitelist are playing.
        """
        active_users_steamids = self.get_active_users_steamids()
        num_nonwhitelisted_users = 0

        for active_user in active_users_steamids:
            player = self.get_playername(active_user, self.whitelist)
            if player is 'Unknown':
                num_nonwhitelisted_users += 1
                nonwhitelisted_playername = self.get_playername(active_user, self.blacklist)
                print('-- nonwhitelisted player: '+str(nonwhitelisted_playername))
        print('>> Total number of non-whitelisted players: '+str(num_nonwhitelisted_users))

if __name__ == '__main__':
    checker = WhiteListVerifier('data/users.txt', 'data/whitelist.csv', 'data/blacklist.csv')
    checker.verify_steamids()

This works, but is essentially O(N**2); you can make it O(N) fairly easily by turning your whitelist/blacklist into dicts:

def __init__(self, users_filepath, whitelist_filepath, blacklist_filepath):
    """
    Get filepaths and load the following files: users file , whitelist file and blacklist file

    users_filepath: Path to a text file which contains the raw output of the 'rcon users' in the in-game console.
    whitelist_filepath: Path to a csv file which contains Steam IDs and player-names of the white-listed players.
    blacklist_filepath: Path to a csv file which contains Steam IDs, player-names and alert-message for the black-listed players.
    """
    with open(users_filepath) as f:
        self.users_rawcontents = f.read()
    raw_whitelist = list(csv.DictReader(open(whitelist_filepath)))
    raw_blacklist = list(csv.DictReader(open(blacklist_filepath)))
    self.whitelist = { p['Steam_ID']: p for p in raw_whitelist }
    self.blacklist = { p['Steam_ID']: p for p in raw_blacklist }

def get_playername(self, steamid, player_records):
    """
    Function to return name of a person if steamid exits in given records
    and return 'Unknown' otherwise.
    """
    return player_records.get(steamid, 'Unknown')

So instead of iterating over the entire list of player_records every time you look someone up, it iterates over your whitelist and blacklist each once at startup-time, making them thereafter be O(1) to search by steamid (which is the only kind of search you ever do).


Delivery geolocations query I have the below MySQL query that does exactly what I need, but I am sure there are better queries that do the same thing.

If you see something very ugly or bad and have any suggestions, I will be very glad to hear them.

SELECT c.`ID_COURIER`, c.`NAME`,
    c.`NICKNAME`, c.`AREA_CODE`,
    c.`PHONE_NUMBER`, d.`ID_DELIVERY`,
    d.`DESCRIPTION`, d.`START_DATE`,
    d.`END_DATE`, d.`CANCELED`,
    r.`LAT`, r.`LNG`,
    (SELECT COUNT(DISTINCT `ID_ORDER`) FROM `ORDERS` WHERE `ID_DELIVERY` = d.`ID_DELIVERY` AND `DELIVERY_DATE` IS NULL AND `CANCELED` = false) as REMAINING_ORDERS
FROM `COURIER` as c
    LEFT JOIN `DELIVERY` as d USING (`ID_COURIER`)
    LEFT JOIN `GEOLOCATIONS` as r ON r.`ID_DELIVERY` = d.`ID_DELIVERY`
WHERE c.`ID_COMPANY` = ? AND c.`DISABLED` = false
    AND (
        CASE WHEN (SELECT MAX(`START_DATE`)
                       FROM `DELIVERY`
                       WHERE `ID_COURIER` = c.`ID_COURIER`)
                 IS NULL THEN d.`START_DATE` IS NULL
                 ELSE d.`START_DATE` = (SELECT MAX(`START_DATE`)
                                        FROM `DELIVERY`
                                        WHERE `ID_COURIER` = c.`ID_COURIER`)
            END )
    AND (
        CASE WHEN (SELECT MAX(`ID_GEOLOCATION`)
                       FROM `GEOLOCATIONS`
                       WHERE `ID_DELIVERY` = d.`ID_DELIVERY`)
                     IS NULL THEN r.`ID_GEOLOCATION` IS NULL
                   ELSE r.`ID_GEOLOCATION` = (SELECT MAX(`ID_GEOLOCATION`)
                                               FROM `GEOLOCATIONS`
                                               WHERE `ID_DELIVERY` = d.`ID_DELIVERY`)
            END)
GROUP BY c.`ID_COURIER`
ORDER BY d.`START_DATE` DESC

Top to bottom, here we go.

COUNT(DISTINCT ID_ORDER)

This DISTINCT aggregation seems superfluous, if ID_ORDER is a true identity column then there should not be duplicates. This will improve performance.
LEFT JOIN DELIVERY AS d USING (ID_COURIER)

I'm not a big fan of the USING keyword as I find it ambiguous. Agreed it's a useful shorthand but to the next person going back to read your code they would have to go and check that both tables have a ID_COURIER column. Plus if one of the columns got renamed at any point it would be more difficult to find the bug.

My suggestion:

LEFT JOIN `DELIVERY` AS d ON c.ID_DELIVERY = d.ID_DELIVERY

WHERE c.ID_COMPANY = ?

This doesn't seem to have any purpose I'm sure you could take it out with no difference.
CASE WHEN (SELECT MAX(START_DATE) and the rest of the statement.

This I find very unusual and not optimal for performance. I'm not sure I completely understand what you are trying to do but I'll give it my best shot. You may want to instead build this into your select. Here is what I suggest for the complete script. I made some edits to the layout to make it easier to read. If I misunderstood the intention of your CASE statements let me know and I can revise.

SELECT
    c.ID_COURIER,
    c.NAME,
    c.NICKNAME,
    c.AREA_CODE,
    c.PHONE_NUMBER,
-- This will return NULL if ID_GEOLOCATION is NULL
    (SELECT MAX(ID_GEOLOCATION) FROM GEOLOCATIONS) AS 'ID_GEOLOCATION',
    d.DESCRIPTION,
-- This will return NULL if START_DATE is NULL
    (SELECT MAX(START_DATE) FROM ORDERS) AS 'START_DATE',
    d.END_DATE,
    d.CANCELED,
    r.LAT,
    r.LNG,
-- Subquery JOIN moved to FROM and WHERE statements
    COUNT(ID_ORDER) AS 'REMAINING ORDERS'
FROM COURIER AS c
    LEFT JOIN DELIVERY AS d ON c.ID_DELIVERY = d.ID_DELIVERY
    LEFT JOIN GEOLOCATIONS AS r ON r.ID_DELIVERY = d.ID_DELIVERY
    LEFT JOIN ORDERS AS o ON d.ID_DELIVERY = o.ID_DELIVERY
WHERE c.DISABLED = false
AND o.DELIVERY_DATE IS NULL AND o.CANCELED = false
GROUP BY c.ID_COURIER
ORDER BY d.START_DATE DESC;