Using PostgreSQL effectively with a banking app I'm kind of newbie for these things but I did something and I want to know how I can do this effectively.

import java.sql.Connection;
import java.sql.Statement;
import java.sql.ResultSet;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Random;

public class BankingApp
{
    public static void main(String[] args)
    {
        Registeration myreg = new Registeration("vivian","stuff","root@root.org");
        myreg.setDatabaseInfo("JavaAppFirst","root","134679");
        myreg.beRegister();
    }
}

interface SQLConnection
{
    public boolean setConnection() throws Exception; // implement the sql connection
    // public void closeConnection(Connection c,Statement stmt) throws SQLException; // close connection
}

class Registeration implements SQLConnection
{
    private int id;
    private String usr,pwd,email;
    private Connection connect;
    private Statement state;
    private String dbName,dbUsername,dbPassword;

    private void tempRegister() throws Exception
    {
        connect.setAutoCommit(false);
        state = connect.createStatement();
        String sqlQuery = "INSERT INTO users (id,uname,upass,uemail) VALUES ('"+this.id+"','"+this.usr+"','"+this.pwd+"','"+this.email+"');";
        state.executeUpdate(sqlQuery);
        state.close();
        connect.commit();
        connect.close();
    }

    public void beRegister()
    {
        try
            {
                if( setConnection() != true )
                    {
                        System.err.println("[+] Connection Error");
                        return;
                    }
                tempRegister();
                System.out.println("[+] Registeration Successfully");
            }catch(Exception e)
            {
                System.err.println(e.getClass().getName()+": "+e.getMessage());
                System.exit(0);
            }
    }

    public boolean setConnection() throws Exception
    {
        connect = null;
        state = null;
        Class.forName("org.postgresql.Driver");
        connect = DriverManager.getConnection("jdbc:postgresql://localhost:5432/"+this.dbName,this.dbUsername,this.dbPassword);
        if( connect == null ) return false;
        return true;
    }

    public void setDatabaseInfo(String a,String b,String c)
    {
        this.dbName = a;
        this.dbUsername = b;
        this.dbPassword = c;
    }

    public Registeration()
    {
        //Random random = new Random();
        id = 0; // random.nextInt(2000);
        usr = null;
        pwd = null;
        email = null;
    }
    public Registeration(String usr,String pwd,String email)
    {
        Random random = new Random();
        this.id = random.nextInt(1000);
        this.usr = usr;
        this.pwd = pwd;
        this.email = email;
    }
}

In my view this code is horrible. How can I develop this code?

this looks really ugly from a readability/formatting point of view.

    public void beRegister()
    {
        try
            {
                if( setConnection() != true )
                    {
                        System.err.println("[+] Connection Error");
                        return;
                    }
                tempRegister();
                System.out.println("[+] Registeration Successfully");
            }catch(Exception e)
            {
                System.err.println(e.getClass().getName()+": "+e.getMessage());
                System.exit(0);
            }
    }

You should use consistent Egyptian (Java) bracing for this along with making sure that you leave some space around the Braces and not around the parenthesis.

Key words also like their space from parenthesis.

Look how much cleaner this little piece of code looks now.

public void beRegister() {
    try {
        if (setConnection() != true) {
            System.err.println("[+] Connection Error");
            return;
        }
        tempRegister();
        System.out.println("[+] Registeration Successfully");
    } catch (Exception e) {
        System.err.println(e.getClass().getName()+": "+e.getMessage());
        System.exit(0);
    }
}


A puzzle about how to write more clean and understandability code private function getFeatureByParam ($classname, $module, $action) {
    $needFeature = array();
    foreach ($this->featureConf as $featureIndex => $feature) {
        if ($feature['classname'] == $classname &&
            $feature['module'] == $module &&
            $feature['action'] == $action) {
                $needFeature = $feature;
                break;
        }
    }
    return $needFeature;
}
private function getFeatureByParam ($classname, $module, $action) {
    foreach ($this->featureConf as $featureIndex => $feature) {
        if ($feature['classname'] == $classname &&
            $feature['module'] == $module &&
            $feature['action'] == $action) {
                return $feature;
        }
    }
    return array();
}

The first style of writing is more understandability.

The second style of writing is more clean and efficient.

But witch one is more better? (Or they are too bad....)

Personally, option 2.

In option 1, by breaking out of the loop, you have to look down until after the end to see what happens, just to see a return.
In option 2, it's obvious what you want and when you get it. Although, I think the condition is a little complex.

There are, of course, other options:

private function getFeatureByParam ($classname, $module, $action) {
    foreach ($this->featureConf as $featureIndex => $feature) {
        if ($feature['classname'] != $classname) continue;
        if ($feature['module'] != $module) continue;
        if ($feature['action'] != $action) continue;

        return $feature;
    }
    return array();
}

private function getFeatureByParam ($classname, $module, $action) {
    foreach ($this->featureConf as $featureIndex => $feature) {
        if ( featureMatches( $feature, $classname, $module, $action ) ) {
            return $feature;
        }
    }
    return array();
}

I'm inclined to option 4 :)


Recursive Scala Case class Reduction I have a case class that is recursive and looks like this:

case class Config(name: String, isEnabled: Boolean, elems: Map[String, MyCase])

case class MyCase(
  id: String,
  isActive: Boolean,
  elems: Option[Map[String, MyCase]])

Where the Config contains the id of the MyCase entries contained as a Map. I have to iterate over this structure and come up with a Map that contains the parent child relations. Say for example., if I have the Config class represented as below (for simplicity, I have just given the id's):

Config(Map("5" -> myCase1, "6" -> myCase2))

5 - Config
 1
  1
  2
 2
6
 1

where id's 5 and 6 are top level entries which in turn has a recursive structure. I have to now come up with a Map that contains the parent child relationship for the id's. So for the case above, I will expect a Map that looks like:

Map(
 "5" -> Seq("5.1"),
 "5.1" -> Seq("5.1.1", "5.1.2"),
 "5.1.1" -> Seq.empty[String],
 "5.1.2" -> Seq.empty[String],
 "6" -> Seq("6.1"),
 "6.1" -> Seq.empty[String]
)

Notice how I append the parent id to the child.

Given a Seq of such Configs,

val configs = Seq(Config1, Config2)

this is what I have come up with:

def relationsFor(config: Config): Map[String, Seq[String]] = {

    def prepareChildren(parentId: String, acc: Map[String, Seq[String]], subElems: Map[String, MyCase]): Map[String, Seq[String]] = {
      subElems.collect {
        case (id, elem) if elem.isActive =>
          val newAcc = acc ++ Map(parentId -> subElems.keys.map(subElemId => s"$parentId.$subElemId").toSeq)
          if (elem.elems.isDefined) {
            val newParentId = s"$parentId.$id"
            val newAcc1 = newAcc ++ Map(s"$parentId.$id" -> elem.elems.get.keys.map(subElemId => s"$newParentId.$subElemId").toSeq)
            prepareChildren(s"$parentId.$id", newAcc1, elem.elems.get)
          } else {
            acc ++ Map(s"$parentId.$id" -> Seq.empty[String])
          }
      }
    }.flatten.toMap

    configs.collect {
      case config if config.isEnabled =>
        config.elems.collect {
          case (parentId, elemCase) if elemCase.elems.isDefined =>
            prepareChildren(parentId, Map.empty[String, Seq[String]], elemCase.elems.get)
          case (parentId, _) =>
            Map(parentId -> Seq.empty[String])
        }
    }.flatten.flatten.toMap
  }

Could this be simplified ? I have already tested this and it works as expected, but I find it a little bit hard to understand. Can this be made much more elegant? I mean can the relationsFor method be made more elegant ?

I tried do it a different way, when you asked this question on StackOverflow.. You find my version below, first I'll try to give you some feedback on your version :

I see you regularly use isDefined together with pattern matching or if / else. I would replace this by matching for Some / None or even better with a combination of map and getOrElse or with fold :

o.map(f).getOrElse(b)
o.fold(b)(f)

These are more ideomatic ways to handle an Option and unwrap the Option for you where you want to use it (calling get on an Option is something you should try to minimize).
It seems like you could have reused newParentId in the places where you wrote s"$parentId.$id", this would mean that newParentId should be placed outside the if else.
I am not sure if you can change the signature of MyCase, but I am not sure what extra the Option brings for elems, but this may be moot if you want to differentiate between None and Map.empty[String, MyCase]], if you don't then no collection or an empty collections is pretty much the same.

Here's what I came up with :

def getMapWithChildIds(config: Config): Map[String, Seq[String]] = {
  def mkId(prefix: String, key: String): String =
    if (prefix.isEmpty) key else prefix + "." + key

  def go(map: Map[String, MyCase], prefix: String): Map[String, Seq[String]] =
    map.flatMap { case (key, elem) =>
      val fullKey = mkId(prefix, key)
      // this will handle an unactive MyCase the same way as a MyCase with no elems
      Option(elem).filter(_.isActive).flatMap(_.elems).fold(
        Map(fullKey -> Seq.empty[String]))(
        submap => {
          val directSubKeys = submap.keys.map(mkId(fullKey, _)).toList
          // map entry for this key + entries for sub keys
          Map(fullKey -> directSubKeys) ++ go(submap, fullKey)
        })
    }

  go(config.elems, "")
}

Which gives for :

val noMap: Option[Map[String, MyCase]] = None

val config =
  Config("a", true,
    Map(
      "5" -> MyCase("id?5", true,
        Some(Map(
          "1" -> MyCase("id?5.1", true,
            Some(Map(
              "1" -> MyCase("id?5.1.1", true, noMap),
              "2" -> MyCase("id?5.1.2", true, noMap)
            ))
          ),
          "2" -> MyCase("id?5.2", true, noMap)
        ))
      ),
      "6" -> MyCase("id?6", true,
        Some(Map(
          "1" -> MyCase("id?6.1", true, noMap)
        ))
      )
    )
  )

The following result :

val map = getMapWithChildIds(config)
// Map[String,Seq[String]] = Map(5.2 -> List(), 5.1.2 -> List(), 5 -> List(5.1, 5.2), 5.1.1 -> List(), 6 -> List(6.1), 6.1 -> List(), 5.1 -> List(5.1.1, 5.1.2))

map.toList.sortBy(_._1).foreach(println)
// (5,List(5.1, 5.2))
// (5.1,List(5.1.1, 5.1.2))
// (5.1.1,List())
// (5.1.2,List())
// (5.2,List())
// (6,List(6.1))
// (6.1,List())


Python script to download adult comics from 8muses I made a simple python script using the BeautifulSoup and Selenium to automatically download adult comics from 8muses.  I've used selenium for the reason that the website uses javascript to load the images.

Enter the the gallery url and download location to start downloading.
Sample gallery urls:

https://www.8muses.com/comics/album/MilfToon-Comics/Milfage/Issue-1
https://www.8muses.com/comics/album/MilfToon-Comics/Lemonade/Lemonade-1

I would like to know improvements to the code or alternative methods to make it work faster. Thanks !

Code : app.py

import os
from multiprocessing.dummy import Pool
from queue import Queue
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
from threading import Thread
import urllib.request
import requests
import shutil

options = Options()
options.headless = True
chrome_driver_path = r"C:\Users\NH\PycharmProjects\SeleniumTest\drivers\chromedriver.exe"
base_url = "https://www.8muses.com"

def fetch_image_url(url,filename,download_location):
    driver = webdriver.Chrome(chrome_driver_path, chrome_options=options)
    driver.get(url)
    page = driver.page_source
    soup = BeautifulSoup(page,"lxml")
    image_url = "http:"+soup.find("img",{"class":"image"})['src']
    download_image(image_url,filename,download_location)

def download_image(image_url,filename,download_location):
    r = requests.get(image_url,stream=True, headers={'User-agent': 'Mozilla/5.0'})
    if r.status_code == 200:
        with open(os.path.join(download_location,str(filename)+".png"), 'wb') as f:
            r.raw.decode_content = True
            shutil.copyfileobj(r.raw, f)
    print("Downloaded page {pagenumber}".format(pagenumber=filename))

if __name__=="__main__":
    print("Album Url : ")
    album_url = input()
    print("Download Location : ")
    download_location = input()
    driver = webdriver.Chrome(chrome_driver_path, chrome_options=options)
    print("Loading Comic...")
    driver.get(album_url)
    album_html = driver.page_source
    print("Comic successfully loaded")
    soup = BeautifulSoup(album_html,"lxml")
    comic_name = soup.find("title").text.split("|")[0].strip()
    download_location = os.path.join(download_location,comic_name)
    os.mkdir(download_location)
    print("Finding comic's pages")
    images = soup.find_all("a",{"class":"c-tile t-hover"})
    page_urls = []
    pages = []
    threads = []
    for image in images:
        page_urls.append(base_url + image['href'])
    print("Found {} pages".format(len(page_urls)))
    for i in  range(len(page_urls)):
            pages.append((page_urls[i],i,download_location))
    p = Pool(3)  # 3 threads in the pool
    p.starmap(fetch_image_url,pages)
    p.close()
    p.join()
    driver.quit()
    print ("DONE ! Happy Reading ")

Github for the project : https://github.com/ggrievous/8muser

There's no reason to use selenium here. Instead of reaching for selenium, first try the easier path. Load the page without javascript and see if you can find any helpful information about how the images get there. There must be something (perhaps AJAX) that gets the URLs for them. They don't just magically appear!

It turns out if you do this, you'll see there isn't any fancy JS stuff. The images are there, they just look like this:

This means, you can extract all these URLs with a single line of BeautifulSoup:

urls = [img['data-src'] for img in doc.find_all('img', class_='lazyload')]

Now some comments about your code:

PEP8 it! Your spacing is inconsistent. Make good use of vertical whitespace. Phrase things like you would paragraphs. That makes things much easier to read.
You don't need selenium, but you definitely shouldn't hard code a driver path in a project that you open source. How many people have a selenium driver exactly at C:\Users\NH\PycharmProjects\SeleniumTest\drivers\chromedriver.exe on their computer?
Nice use of functions to separate concerns
You should probably use BeautifulSoup(page, 'html5lib') instead of lxml
Your construction of image_url is a bit sloppy. Typically, we'd reach for urllib.path to build paths instead of just doing string concatenation.
Use pathlib instead of os.path
'Mozilla/5.0' isn't a User-Agent that's going to fool anybody. If you're really trying to stay under the radar, use a real UA
But none of that matters, because you appear to request pages as fast as possible. Add sleep()s in between downloading. Throttle your scraper.
threading is a bit useless in Python. This is somewhat of an I/O bound task (for which threads are well suited), but the HTML parsing and extraction could definitely be done concurrently with web requests (but threading doesn't allow this). You almost always want to reach for multiprocessing.
Use the context manager of a pool instead of manually calling close() and join():

with Pool() as pool:
    pool.imap_unorderd(fetch_image_url, pages)

Also, don't pass a parameter to Pool. It defaults to the number of CPU cores, which is almost always what you want
starmap is ordered and blocking. It only can process things in order. This is okay in this case, because you aren't actually returning anything, but if you were say doing math you probably want imap_unordered which yields results as they arrive (likely out of order).
Don't print from a separate process. You want a single process writing to stdout, otherwise you can have write contention (which you'll luck out and probably never run into because your strings probably fit inside the stdout buffer, but it's possible they may not under certain circumstances).

But this all culminates in the following advice: don't use Python to download things!

Especially since this scraping doesn't appear to be useful as a library (instead, it seems like you just are providing a CLI utility for a human to download these things). Given this, it's much smarter and safer to not reinvent the wheel. There are tools that already do jobs like this well: namely, wget (it appears that you're on windows, you can and should use the Ubuntu subsystem, which will have wget). wget is particularly suited for this job and has tons of builtin functionality which will be super useful to you. This includes:

Not redownloading things
Throttling (including random delays)
Restarting after a catastrophic (program crashing) failure
Retrying requests per HTTP spec

All of these are things that your script doesn't do currently. In particular, in python it's very easy to do something like this:

pages = download_hundreds_of_pages()  # takes hours...
for page in paages:  # oops, this NameErrors and you lose everything you've downloaded
  pass

Mistakes like this are too easy to make. You can completely avoid them with the following workflow:

Build up a list of urls you want to download (perhaps with python)
Use wget -nc -i urls.txt to download them
Repeat as necessary

For you, that would involve making a list of urls containing the images. Then do wget -nc -i pages.txt. That will download all of the pages to the current directory. Then you can make a Python script which uses beautiful soup (and the line I mentioned above) to extract the image urls: python3 extract_image_urls.py > image_urls.txt. Then to download them do wget -nc -i image_urls.txt. If your python script fails at any point, you don't lose all of the downloads you've already done. You can wrap all of this in a convenient bash script.


Checkbox binding to change the ListView's items source I am using a checkbox to filter some components on my ListView by changing the variable's content which is bound to this ListView:

C#

#region Constructor

void CreateAllParts()
{
    // GetParts() returns List<Part>. Now we can pour it's results
    // using the following select in List<EditPartViewModel> on the
    // fly.
    // allpvms: All Part View Models.
    allpvms = _partRepository.GetParts().Select(p => new EditPartViewModel(p)).ToList();

    foreach (EditPartViewModel cvm in allpvms)
    {
        cvm.PropertyChanged += this.OnPartViewModelPropertyChanged;
    }

    this.AllParts = new ObservableCollection<EditPartViewModel>(allpvms);
    this.AllParts.CollectionChanged += this.OnCollectionChanged;
}

void LoadErrorParts()
{
    if (_errorParts)
    {
        this.AllParts = new ObservableCollection<EditPartViewModel>(allpvms.Where(p => p.Price == null));
        this.AllParts.CollectionChanged += this.OnCollectionChanged;
    }
    else
    {
        this.AllParts = new ObservableCollection<EditPartViewModel>(allpvms);
        this.AllParts.CollectionChanged += this.OnCollectionChanged;
    }
}

#endregion // Constructor

#region Public Interface

/// <summary>
/// Returns a collection of all the EditPartViewModel objects.
/// </summary>
public ObservableCollection<EditPartViewModel> AllParts
{
    get
    {
        return _allParts;
    }
    set
    {
        _allParts = value;
        OnPropertyChanged("AllParts");
    }
}

public bool ErrorParts
{
    get
    {
        return _errorParts;
    }
    set
    {
        _errorParts = value;
        OnPropertyChanged("ErrorParts");
        LoadErrorParts();
    }
}

#endregion // Public Interface

XAML

<CheckBox DockPanel.Dock="Top"
          IsChecked="{Binding ErrorParts, Mode=TwoWay}"
          HorizontalAlignment="Right"
          Content="Parts with Error"
          />

<ListView
  AlternationCount="2"
  d:DataContext="{d:DesignInstance Type=vm:MockViewModel, IsDesignTimeCreatable=True}"
  ItemsSource="{Binding AllParts}"
  >
  <ListView.ItemContainerStyle>
    <Style TargetType="{x:Type ListViewItem}">
      <EventSetter Event="MouseDoubleClick" Handler="ListViewItem_MouseDoubleClick" />
      <Setter Property="HorizontalContentAlignment" Value="Stretch"/>
    </Style>
  </ListView.ItemContainerStyle>
</ListView>

ListViews content get collected by a DataTemplate in the Resources.

Is the way I use the binding concept look okay? Any corrections or "good practice" methods I should consider when doing similar procedures?

You should probably bind ItemsSource to CollectionView, instead of binding to ObservableCollection. This way you don't have to recreate collection every time you want to apply your filter. May look like this:

//constructor
Parts = new ObservableCollection<EditPartViewModel>(allpvms);
PartsCollectionView = (ListCollectionView)CollectionViewSource.GetDefaultView(Parts);

...............

public ListCollectionView PartsCollectionView { get; private set; }

public bool ErrorParts
{
    get
    {
        return _errorParts;
    }
    set
    {
        _errorParts = value;
        OnPropertyChanged("ErrorParts");
        PartsCollectionView.Filter = value ? (Predicate<Object>)(p => ((EditPartViewModel)p).Price == null) : null;
    }
}

Also if your viewmodles can contain errors, you should probably move your error-checking logic inside those viewmodels. So instead of writing p.Price == null you would write p.HasErrors or something.

You should also indicate, that ErrorParts is actually a bool property, unlike AllParts which is a collection. DisplayErrors or ShowErrors or FilterErrors etc. are better names for this property in my opinion.


Implementation of a new algorithm for sklearn In the Python library, sklearn is implemented the algorithm for SparsePCA.

I have written the code for a another version of this algorithm that is much faster in some situations.  I have not enough experience with Python and sklearn in order to write code that could be loaded in the sklearn repositories.

It would be nice if you can help me to write this code in order to make it 100% compatible with the sklearn standards. If you have experience with machine learning would be nice if you help me to consider bad situations that I may have not considered in order to make the algorithm more robust.

import numpy as np
import sklearn as sk
import pandas as pd
from sklearn.decomposition import SparsePCA, PCA

import random

class IterativeSPCA():

    def __init__(self, Npc=None, alpha=1, max_iter=5000, tol=1e-8):
        self.Npc = Npc #number of components
        self.alpha = alpha  # penalty value
        self.max_iter = max_iter # max number of iteration
        self.tol = tol # accepted tollerance error

    def spca_iterate(self, u_old, v_old):
        # this is the equivalent of the nipals algorithm but designed in order to
        # obtain a sparse loading
        v_new = np.zeros(shape=v_old.shape)
        u_new = np.zeros(shape=u_old.shape)

        for iteration in range(self.max_iter): #repeat the iteration non more than max iter
            y = np.dot(self.X.T, u_old).squeeze()
            v_new = (np.sign(y)*np.maximum(np.abs(y)-self.alpha, 0)).squeeze()
            norm_v = np.linalg.norm(v_new)**2
            if norm_v == 0: #if norm v is 0 it means that no components have been selected. So you should reduce the penalty lambda

                break

            x_times_v = np.dot(self.X, v_new)
            x_times_v.shape = (self.m, 1)
            u_new = x_times_v/np.linalg.norm(x_times_v)

            if np.linalg.norm(v_new)==0:#check again if the norm is 0
                break
            if np.linalg.norm(v_new - v_old)<self.tol or np.linalg.norm(-v_new - v_old) < self.tol:#check if there is convergence
                break
            u_old = u_new
            v_old = v_new

        if iteration == self.max_iter-1:
            print("No Convergence. Error!!!")

        norm_v = np.linalg.norm(v_new)
        v_new.shape = (self.n, 1)
        v_final = sk.preprocessing.normalize(v_new, axis=0)
        u_final = u_new * norm_v
        return v_final, u_final

    def fit(self, X):

        self.m, self.n = X.shape
        self.components_ = np.zeros(shape=(self.n,self.Npc))
        self.sT = np.zeros(shape=(self.m,self.Npc))
        self.X = X

        for i in range(self.Npc):

            pca = PCA(n_components=1).fit(self.X)
            v = pca.components_
            v = sk.preprocessing.normalize(v)
            u = self.X.dot(v.T)
            s = np.linalg.norm(u)

            u_old = u
            v_old = v*s

            v_final, u_final = self.spca_iterate(u_old=u_old, v_old=v_old)
            self.sT[:, i] = u_final.squeeze()
            self.components_[:, i] = v_final.squeeze()
            u_final.shape = (self.m, 1)
            self.X = self.X-np.dot(u_final, v_final.T)
        return self.components_, self.sT

print("start")
X = np.random.random((100, 20)) #generate a random matrix
X = sk.preprocessing.scale(X)   # scale the data
alpha = 1 # set the penalty
nPCs = 2 # set the number of components

import time

start_time = time.time()
a = IterativeSPCA(Npc=nPCs, alpha=alpha)
sP, sT = a.fit(X)
time1 = time.time()

spca = SparsePCA(n_components=nPCs, alpha=alpha, ridge_alpha=0)
spca.fit(X)
time2 = time.time()

import pandas as pd
df = pd.DataFrame(columns=['it_spc1','spc1'])
df['it_spc1'] = a.components_[:,0]
df['spc1'] = spca.components_.T[:,0]
df['it_spc2'] = a.components_[:,1]
df['spc2'] = spca.components_.T[:,1]
print("time iterative SPCA=", time1-start_time, "time SparsePCA=", time2-time1)
print("finish!")

Thank you for posting this.

You didn't include any result timings. CPU performance and error performance on an available dataset would be of interest, for evaluating against sklearn's baseline code. Highly correlated inputs would be more interesting than uniform random numbers.

Noticing the import just above it makes the class name perfectly clear, but nonetheless please give it a docstring explaining:

"""Iterative Sparse PCA (principal component analysis)."""

Please fix a comment so it reads:

    self.Npc = Npc  # number of principal components

or, better, put it in a docstring. Consider deleting the redundant max_iter comment, or perhaps rephrase it to describe max number of principal components if that is its function.

Typo: tolerance

def spca_iterate(self, u_old, v_old):

The public API should probably talk about u_init & v_init.

Typo: no more (but consider deleting this redundant comment)

    v_new = np.zeros(shape=v_old.shape)
    u_new = np.zeros(shape=u_old.shape)

This pair of assignments appears to have no effect.

        if norm_v == 0:  # if norm v is 0 it means that no components have been selected. So you should reduce the penalty lambda
            break

This seems to be speaking about this author's alpha in terms of another author's lambda. Your docstring should definitely mention the effect of alpha on the solution's sparsity.

        if np.linalg.norm(v_new)==0:#check again if the norm is 0
            break
        if np.linalg.norm(v_new - v_old)<self.tol or np.linalg.norm(-v_new - v_old) < self.tol:#check if there is convergence
            break

The first test is pointless (redundant). Please delete it. The 2nd test is phrased oddly. Please express it as:

        if abs(np.linalg.norm(v_new - v_old)) < self.tol:  # check for convergence
            break

This assignment:

        x_times_v.shape = (self.m, 1)

and the v_new .squeeze(), make me concerned that we didn't take an opportunity to fix up shapes before the loop began. The squeeze returns either a view or the underlying object. A single view would be just fine, but a chain of max_iter views would be a problem. Can we maybe banish the shape concerns to the init code at top of function, outside the loop?

        print("No Convergence. Error!!!")

A raise might be more appropriate here.

    norm_v = np.linalg.norm(v_new)

We were squaring before. This statement might be fine, but it warrants a comment, as one norm_v is not like the other. Or, put another way, the whole method really really needs a specific literature reference describing the method.

It's not actually clear what the squaring was about. If FP negative zero was a concern, then abs() should suffice.

fit() would like a docstring, please.

    self.components_ = np.zeros(shape=(self.n,self.Npc))
    ...
    self.components_[:, i] = v_final.squeeze()

This made no sense to me. It is conventional to use underscore suffix to avoid conflict with pre-existing symbols (e.g. min_) and prefix for private variables. But here it appears you just wanted a local variable (not an attribute) named components. For that matter, it's not clear what's going on with m, n, sT, & X. It appears they should be locals.

X = np.random.random((100, 20)) #generate a random matrix
X = sk.preprocessing.scale(X)   # scale the data

Please delete both remarks.

alpha = 1  # set the penalty

Good. Very clear, consistent with the literature.

nPCs = 2  # set the number of components

You felt a need to write a comment. This suggests nPCs was misnamed, and would be clearer as num_components. Similarly for Npc in the ctor.

Consider using a private _spca_iterate name.

The a name helps nobody. Consider turning

a = IterativeSPCA(Npc=nPCs, alpha=alpha)
sP, sT = a.fit(X)

into

sP, sT = IterativeSPCA(Npc=nPCs, alpha=alpha).fit(X)

Maybe a single fit() function, with no class, would suffice.

The sP and sT names could be more helpful. I understand that P denotes principle components, but you didn't offer a literature reference so T is a bit obscure, it doesn't seem to relate to "feature".

df['it_spc1'] = a.components_[:,0]

What is going on here? Didn't you just put that in sP, but now you're unwilling to use it?

df['spc1'] = spca.components_.T[:,0]

Oh, I see. You were going for API compatibility. I guess you know already how I feel about SPCA fit() being void. I cannot fathom why you return transpose of what SPCA offers.

Ok, maybe .components_ is an important part of your public API, at least for backward compatibility. But it seems like you could be friendlier to callers, so it is easier to consume your API and correctly interpret results.


Critique of FilterInputStream and FilterOutputStream classes I'm working my way through The Java Programming Language, Fourth Edition - The Java Series.  This is Exercise Exercise 20.3:

  Create a pair of Filter stream classes that encrypt bytes using any
  algorithm you choose—such as XORing the bytes with some value—with your
  DecryptInputStream able to decrypt the bytes that your EncryptOutputStream
  class creates.

Is the following an adequate solution? I understand that the key should be randomly generated and at least as long as the message. They key used is just for illustration.

EncryptOutputStream.java

import java.io.FilterOutputStream;
import java.io.IOException;
import java.io.OutputStream;

public class EncryptOutputStream extends FilterOutputStream {
    private final static int XOR_KEY = 1;

    public EncryptOutputStream(OutputStream out) {
        super(out);
    }

    @Override
    public void write(int b) throws IOException {
        super.write(b ^ XOR_KEY);
    }
}

DecryptInputStream.java

import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;

public class DecryptInputStream extends FilterInputStream {
    private final static int XOR_KEY = 1;

    protected DecryptInputStream(InputStream in) {
        super(in);
    }

    @Override
    public int read() throws IOException {
        int b = super.read();
        return (b != -1 ? b ^ XOR_KEY : b);
    }
}

EncryptionTest.java

import org.junit.Test;
import static org.junit.Assert.*;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.FilterInputStream;
import java.io.FilterOutputStream;
import java.io.InputStream;
import java.io.OutputStream;

public class EncryptionTest {
    @Test
    public void testEncryptOutputStream() throws Exception {
        String testStr = "The quick brown fox jumps over the lazy dog.";
        InputStream in = new ByteArrayInputStream(testStr.getBytes("UTF-8"));
        OutputStream out = new ByteArrayOutputStream();
        FilterOutputStream encryptOut = new EncryptOutputStream(out);

        int b;
        while ((b = in.read()) != -1) {
            encryptOut.write(b);
        }

        assertEquals("Uid!pthbj!csnvo!gny!ktlqr!nwds!uid!m`{x!enf/", out.toString());

        in.close();
        encryptOut.close();
    }

    @Test
    public void testDecryptInputStream() throws Exception {
        String testStr = "Uid!pthbj!csnvo!gny!ktlqr!nwds!uid!m`{x!enf/";
        InputStream in = new ByteArrayInputStream(testStr.getBytes("UTF-8"));
        FilterInputStream decryptIn = new DecryptInputStream(in);
        OutputStream out = new ByteArrayOutputStream();

        int b;
        while ((b = decryptIn.read()) != -1) {
            out.write(b);
        }

        assertEquals("The quick brown fox jumps over the lazy dog.", out.toString());

        decryptIn.close();
        out.close();
    }

    @Test
    public void testEncryptDecrypt() throws Exception {
        String testStr = "The quick brown fox jumps over the lazy dog.";
        OutputStream out = new ByteArrayOutputStream();
        InputStream in = new ByteArrayInputStream(testStr.getBytes("UTF-8"));
        FilterOutputStream encryptOut = new EncryptOutputStream(out);

        int b;
        while ((b = in.read()) != -1) {
            encryptOut.write(b);
        }

        assertEquals("Uid!pthbj!csnvo!gny!ktlqr!nwds!uid!m`{x!enf/", out.toString());

        String resultStr = out.toString();
        InputStream inResult = new ByteArrayInputStream(resultStr.getBytes("UTF-8"));
        FilterInputStream decryptIn = new DecryptInputStream(inResult);
        OutputStream outResult = new ByteArrayOutputStream();

        b = 0;
        while ((b = decryptIn.read()) != -1) {
            outResult.write(b);
        }

        assertEquals(testStr, outResult.toString());

        encryptOut.close();
        decryptIn.close();
        outResult.close();
    }
}

It looks like you have covered the bases quite well. There are some things I would like to suggest, and the first one is the most important:

  The performance of Input/Output Streams that read/write 1 byte at a time is really, really bad. You should also override the bulk read(...)/write(...) methods.

Now, when you get around to doing the bulk methods, you will have some challenges with getting the byte manipulation right, but that's another problem.... ;-)

Now in your decrypt routine you have:

int b = super.read();
return (b != -1 ? b ^ XOR_KEY : b);

I don't like this. The reason is that it has too much negative logic..... and even though it relates b to a constant -1 you return b instead of -1. Consider the following alternative:

return b == -1 ? -1 : b ^ XOR_KEY;

I know it is a minor nit-pick, but it does make a difference.

On that note, you had better make sure that XOR_KEY is always positive... a negative XOR_KEY will cause the result to be negative, and break all sorts of InputStream readers.

Now, you say the key should be as long as the message... that's not true. Even some of the best algorithms use 2048-bit keys (256 bytes). and many of them use much less (16 bytes...).

Additionally, you hint that you know that storing the key the way you do is incorrect ... It should be passed in as a constructed value or something, and not hard-coded as a static variable.

What you have is more than 'adequate', it is neat, and appears to be fully accurate... but, in order to be good, or great, it needs to do the bulk-operations...


Search string in a list data = ['HTTP/1.1 200 OK', 'CACHE-CONTROL: max-age=1810', 'DATE: Wed, 14 May 2014 12:15:19 GMT', 'EXT:', 'LOCATION: http://192.168.94.57:9000/DeviceDescription.xml', 'SERVER: Windows NT/5.0, UPnP/1.0, pvConnect UPnP SDK/1.0', 'ST: uuid:7076436f-6e65-1063-8074-78542e239ff5', 'USN: uuid:7076436f-6e65-1063-8074-78542e239ff5', 'Content-Length: 0', '', '']

From the above list, I have to extract the .xml link.

My code:

for element in data:
    if 'LOCATION' in element:
        xmllink = element.split(': ').[1]

It's taking too much time.  How can I make this faster?

Actually I am doing SSDP discovery for finding devices in a network.  After sending the M-SEARCH command, devices send a datagram packet which I have taken in a data variable.  From this I have to extract the file link of that device for processing it.

When I use indexing to extract, it was done quickly.

You want to test for element.startswith('LOCATION: ').  You are doing 'LOCATION' in element, which is not only slower since it has to check at every position of every element, it might also lead to false matches.

Also, element and data are poor names.  I suggest header and headers, respectively.

My suggestion:

LOC = 'LOCATION: '
xmllinks = [header[len(LOC):] for header in headers if header.startswith(LOC)]
if xmllinks:
    xmllink = xmllinks[0]


How to re-factor a common String comparison Code below is scattered all over the code base :

if (StringUtils.contains(manager.getName, "custom")){

}

It just checks if an object attribute contains a predefined String and if it does, enter condition.

Is there a more elegant way of achieving above, perhaps using an enum?

It seems data envy. I'd create nameContains method in the Manager class.

public boolean nameContains(final String searchText) {
    if (StringUtils.contains(name, searchText)) {
        return true;
    }
    return false;
}

Instance variables (fields) should be private. See: why instance variables in java are always private


Write the running time to file without capturing the program's output I want to run an application and write (append) its processing time into a file. Also, I want to see the stdout and stderr output on the shell, but don't want to write it in a file. After running the process, I'd like to see time measured.

Since I'm on Bash, one solution would be this one, but it's pretty ugly, since tail needs to parse the whole output:

function run_time {
    { local TIMEFORMAT=%R; time "${@}"; } 2>&1 | tee /dev/stderr | tail -n 1 >> run_time.txt
}

Another variant would be to use GNU time:

function run_time {
    /usr/bin/time --quiet --format '%e' --append --output=run_time.txt "${@}"
    tail -n 1 run_time.txt
}

When running the function like this, the measured time gets written into the file, of course.

run_time mycommand > target.txt

A work-around would to redirect tail's output to stderr or a sub-shell:

run_time sh -c "exec mycommand > target.txt"

And here's my question: It there a more elegant way?

Kudos

It's good to see appropriate quoting and correct use of "$@" in these shell scripts.  Although the samples are short, it's still good to see nice readable code.

Problems with the first approach

All the output of the command is sent to /dev/stderr; it's not possible to pipe the output into another command, or to separate the output and error streams.
The exit status of the command is lost, so you can't use this function as part of a conditional (if, while, &&, ||, ...).
Only a single command can be timed, rather than a full pipeline.

Problems with the second approach

The exit status of the command is lost.
Only a single command can be timed (but that's always the case with an external time command, and why it's a built-in in Bash).

An improvement

You can retain the exit status when using GNU time, by using a process substitution as the output file:

function run_time {
     command time --quiet \
                  --format '%e' \
                  --output >(tee -a run_time.txt >&2) \
                  "${@}"
}

Notes

I've also used the command keyword to find the external time without hard-coding its path.  You may or may not choose to follow this course.
I've sent the time result to standard error, for consistency with the command and built-in (note that the built-in outputs to the shell's standard error, not that of the pipeline it's timing).

Alternative using built-in

function run_time {
  local TIMEFORMAT='%R'
  { { time "$@" 2>&3; } 2> >(tee -a run_time.txt >&2); } 3>&2
}

That's a bit hard to follow, so let's unpick it from the inside:

{ time "$@" 2>&3; }

The command's standard output is unchanged (stream 1), but its stderr is redirected to stream 3; time's output will be on stream 2.
{...} 2> >(tee ... >&2)

The time output is redirected to the tee command; it writes to the file and to the enclosing stream 2 (the function's standard error stream).
{ ... } 3>2

The saved standard error stream of the timed command is diverted back to the function's standard error.

This allows independent redirection of the timed command's standard output and standard error (plus timing results).  The function's error status is that of the timed command.

One disadvantage is that it doesn't wait for the process substitution to exit before the function returns, so tee's output may occur unexpectedly late (e.g. after Bash has shown the next command prompt, in an interactive shell).


Atomic fixed-size parallel std::vector wrapper I'm in need of a simple wrapper for a fixed-size std::vector that can ensure atomic insertion.

Since my only real problem is with atomic insertion and I know beforehand the amount of entries it will have, I don't need any other parallelism related feature.

I am happy to have the initial allocation (std::vector::resize) run from the main thread, so it shouldn't need anything fancy.

Although there are great, free to use parallel vectors available, I think that my particular use case, being fixed size shouldn't require any fancy library (e.g. TBB).

I was thinking about something along these lines:

template <typename T>
class AtomicFixedSizeVector {
public:

    // a bunch of passthrough methods here...

    inline void reserve(size_t count) {
        m_data.resize(count);
        //m_data.reserve(count);
    }

    inline void push_back(const T& entry) {
        size_t index = m_count.fetch_add(1, std::memory_order_relaxed);
        m_data[index] = entry;
    }

    // I don't think I need this at all, but just in case
    inline size_t size()const {
        return m_count.load(std::memory_order_relaxed);
    }

private:
    std::vector<T> m_data;
    std::atomic<size_t> m_count;
};

It needs to be a wrapper to std::vector since the other parts of code take one.

I thought I'd ask this question as my knowledge of atomics is rather limited, and although it seems ok when executed, it's hard to tell if it's actually going well or I'm being lucky multiple times in a row.

Another question that I have is if I can trust std::vector::reserve to do the allocation on the spot or if it's implementation-dependent. I'm asking because if it allocates on the spot I don't need to override the behavior of std::vector::size and can use a passthrough wrapper there, too.

Please note that this is not a premature optimization but rather a post-mortem one, my insertion could be parallelized but only if it ends up adding onto a single block of entries.

This std::vector block of entries will only store uint32_t in case it helps.

Edit:

The current use case is to insert from multiple threads, but this was designed knowing beforehand that by the time the data is accessed, the threads will have been joined.

There is an upper limit on the amount of values to be inserted so the vector will be preallocated before insertion.

You haven't shown how you intend to use this thing, so it's impossible to say for sure whether it "works correctly." For example, obviously it doesn't work if you're planning to use it like this:

AtomicFixedSizeVector<int> v;
v.reserve(100);
std::thread t1([&]() {
    while (v.size() < 10) v.push_back(42);
});
std::thread t2([&]() {
    while (v.size() < 10) std::this_thread::sleep_for(1ms);
    std::cout << v.m_data[0] << "\n";  // DATA RACE!
});

Obviously it won't compile if your type T isn't default-constructible. You could address that idiomatically by adding

static_assert(std::is_default_constructible_v<T>);

to the top of the class body — thus hopefully reassuring the client-programmer that the crazy template error message they just got was intended, and not a bug.

Your reserve method can only be called once, and m_count increases monotonically. Think about whether to implement a clear() method, and what it might do. Think about whether reserve() should reset m_count to zero or not.

My kneejerk reaction is that reserve shouldn't exist at all, and that you should force the user to construct the container with the right capacity to begin with:

explicit AtomicFixedSizeVector(int capacity);

ALX23z is right that what you mean is FixedCapacity, not FixedSize.


Game leveling system I made this code for simple leveling game.

I think this code may have some issues with security. Also, are there some way to make this code more efficient and reliable?

For level scaling I will use this method. Should I make lvl, xp and nxtlevel variables to array or is that a bad idea?

class LevelingSystem
{

  private $_lvl;
  private $_xp;
  private $_nxtlevel;

  function __construct($lvl=1,$xp=0,$nxtlevel=30)
  {

    $this->_lvl=$lvl;
    $this->_xp=$xp;
    $this->_nxtlevel=$nxtlevel;
  }

  function setxp($setxp)
  {
    global $db;
    $stmt = $db->prepare("UPDATE user_atri SET totalxp =totalxp+".$setxp." WHERE id=1");
    $updatexp=$stmt->execute();
    return $updatexp;
  }

  private function getxp()
  {
    global $db;
    $query = $db->prepare("SELECT totalxp FROM user_atri where id=1");
    $query->execute();
    $totalxp=$query->fetch(PDO::FETCH_ASSOC);
    $this->_xp=$totalxp["totalxp"];
    return $totalxp["totalxp"];
  }

  function lvl()
  {

    $this->getxp();
    while ($this->_xp>=$this->_nxtlevel)
    {
        $this->_lvl+=1;
        $this->_xp =$this->_xp - $this->_nxtlevel;
        $this->_nxtlevel = round($this->_nxtlevel * 1.6);
    }

    //$levelprint=sprintf("Youre level is: %s ",$this->_lvl);
    //echo $this->_xp;
    //echo "<br>";
    //return $levelprint;
    return $this->_lvl;

  }

}

First: in the book 'Refactoring' by Martin Fowler,  one of the code smells is "Obsession with elementary types". And the advice is to create/take out variables and move them to another class.

Array is an elementary type. The resolution is to create an entity class Settings and set member variables $lvl,$xp,$nxtlevel.

Second: global $db; transfer through object.

Using global variables is bad practice. If you want to use a Database you should create an object for Database access. Best practice use DI. For PHP one such library is php-di.

Your code may look like this:

<?php

  class LevelingSystem {

  private $settings;
  private $db;

  function __construct(SettingsEntity $settings, DBInterface $db ){
    $this->settings=$settings;
    $this->db = $db;
  }

  public function setxp($setxp) {
    $result =  $this
                ->db
                ->prepare("UPDATE user_atri SET totalxp =totalxp+" . $setxp . " WHERE id=1")
                ->execute();

    if(!$result) {
       throw new Exception('DB error');  //you can send your exception
    }

    return $this;
  }

  public function calcSettings() {
   $xp = $this->getxp();
   $nxtlevel = $this->settings->getNxtlevel();
   $lvl = $this->settings->getLvl();

   while ($xp >= $nxtlevel) {
        $lvl += 1;
        $xp = $xp - $nxtlevel;
        $nxtlevel = round($nxtlevel * 1.6);
    }

     $this->settings
     ->setlvl($lvl)
     ->setXp($xp)
     ->setNxtlevel($nxtlevel);

     return $this;
  }

  public function getLvl() {
      return $this->settings->getLvl();
  }

  private function getxp() {
    $totalxp = $db
                ->prepare("SELECT totalxp FROM user_atri where id=1")
                ->execute()
                ->fetch(PDO::FETCH_ASSOC);

    if(!isset($totalxp["totalxp"])) {
       throw new Exception('Some DB error'); //you can send your exception
    }

    return $totalxp["totalxp"];
  }
}


Disposing of an SQL dataset I am getting a compiler warning when running my code analysis.  It is CA2202.  It is saying that I have not properly disposed my dataset through all exception paths.  Here is the code:

    private System.Data.DataSet MyDataSet()
    {
        DataSet dat_set = new DataSet();

        using (SqlConnection conn = new SqlConnection(conn_string))
        {
            try
            {
                da_1 = new SqlDataAdapter(sql_string, conn);
                dat_set.Locale = CultureInfo.InvariantCulture;
                da_1.Fill(dat_set, "Table_Data_1");
                MainForm.wasSuccess = true;
            }
            catch (SqlException ex)
            {
                Console.Write(ex.ToString());
            }
        }

        return dat_set;
    }

Any thoughts on how to properly dispose this object?

You don't want to dispose it. Otherwise your client code will be working off a disposed object and you'll be getting ObjectDisposedException's left and right.

The question is, does your client code really need a DataSet? or does it simply need a representation of the data?

If it actually needs to be working off a DataSet object, you can ignore the warning, as long as your client code is properly disposing it when it's done working with it.

If all you need is a representation of the data, then the solution is to not return a DataSet, but some IEnumerable<Foo> or IReadOnlyList<Foo>, where Foo is a Plain Old CLR Object (POCO - a simple class exposing nothing more than property getters and setters).

Instead of using a DataSet, you could use a SqlDataReader, which you obtain by executing a SqlCommand - note that returning an IEnumerable<T> makes it possible to yield return your results:

public IEnumerable<Foo> GetData(string connectionString, string sqlStatement)
{
    using (var connection = new SqlConnection(connectionString))
    using(var command = new SqlCommand(sqlStatement, connection))
    {
       connection.open();
       using (var reader = command.ExecuteReader())
       {
            while (reader.Read())
            {
               yield return new Foo
                   {
                       Bar = (int)reader[0], // retrieve by field index, or..
                       Something = (string)reader["SomeField"] // retrieve by field name
                   };
            }
       }
    }
}

Couple of other points:

Don't use underscores in identifiers. Use the more standard camelCase instead.
Don't abbreviate or disemvowel identifiers. There's no reason to do this.
Don't update the UI in a method whose responsibility is to retrieve data.
If you don't know what to do with an exception, let it bubble up, the client code is more likely to know what to do (e.g. pop a messagebox, or show a meaningful error message in the UI).
Use objects. The instruction MainForm.wasSuccess = true; looks very very much like you're accessing a static member, which is bad because static things belong to a type, as opposed to an instance.


Determining if an answer can be generated How can I improve my code below for the following question?  While it works, I am not satisfied with how it looks; I was hoping I can somehow keep it recursive, and have the ans filled up inside the function, and reduce the number of return statements that I have.  Treat this like an interview problem.

Worst case: \$O(2^n)\$? Please verify and elaborate

Space complexity: \$O(1)\$

  Problem:

  You have four integers, a, b, c, d, and the goal is to make a == c and
  b == d, given that you are only allowed to do a+b, b, or a, b+a at any
  given transformation.

  For example, if a = 1, b = 2, c = 3, and d = 8
  you can start with (1, 2), change it to (3, 2), change that to (3, 5),
  and then change that again to (3, 8) to succeed.

  static int ans = 0;
  public static void ablehelper(int a, int b, int c, int d){
      if(a != c && (b + a) > c){
          return;
      }
      if(b != d && (b + a) > d){
          return;
      }
      if(a == c && b == d){
          ans = 1;
          return;
      }
      ablehelper(a + b, b, c, d);
      ablehelper(a, b + a, c, d);

  }
  public static String able(int a, int  b, int c, int d){
      ablehelper(a, b, c, d);
      if(ans == 1){
          return "Able to generate";
      }else{
          return "Not table to generate";
      }
  }

Update:

As per one of the algorithms described below, I wrote this:

public static String betterSolution(int a, int b, int c, int d){
          while( c > a || d > b){
              if(c > d){
                  c = c-d;
              }else{
                  d = d-c;
              }
          }
          if( c == a &&  d == b){
              return "Able to generate";
          }else{
              return "Not able to generate";
          }
      }

This is an interesting problem! You kept our chatroom quite occupied with discussing this question for quite a while :)
Your approach is interesting but it has one big flaw:
You're starting from the wrong direction!
Instead of starting from a and b, start from c and d. Ask yourself this question: What are the ways to reach that number?
Consider this problem: (I will use the notation a, b --> c, d)
4, 5 --> 15, 19

What has to be the step before 15, 19? Which number was added to which?
Was it 15, x or was it x, 19? x, 19 would be nuts as 15 is less than 19. Therefore, we know that it was 15, x. And x must be 4, as 15 + 4 = 19.
By continuing to subtract the biggest number with the lowest number, we can find out all the possible inputs that can produce these two numbers.
15, 19
15, 4
11, 4
7, 4
3, 4
3, 1
2, 1
1, 1

As a, b was 4, 5, we can see already at 15, 4, that this is not possible because 4 < targetB.
Doing this for your example of 1, 2 --> 3, 8:
3, 8 <--- 8 is bigger, so decrease it by 3
3, 5 <--- 5 is bigger, so decrease it by 3
3, 2 <--- 3 is bigger, so decrease it by 2
1, 2 <--- 2 is bigger, so decrease it by 1
1, 1

Wait, what's that!? 1, 2 is in the list!? It's our lucky day! We've found a matching a, b! So this will return true.
I will leave the fun part of implementing the code up to you :) I hope your understand this approach.
As for the time complexity: Significantly faster than your current approach. As for the real time complexity, ask @rolfl.


"Hello world!" in Prolog I have just written the following "hello world" program in Prolog. Although it is as simple as it can be, I would like your feedback as Prolog is very new to me.

#!/usr/bin/swipl -q -t main -f

main :-
    write("Hello world!"), nl, fail.

Questions

Shebang: I am used to use #!/usr/bin/env python in Python. This is the
prefered method, because it does not rely (so much) on a specific system.
write: it seems as if I could also use format. Are there other options?
Style guide: I have not found any style guide regarding indentation or spaces after ,.
Is quitting with fail to usual way? I have seen halt, too.
File ending: I saved it as helloworld.pro. What file extensions are common?

As the code is short and I have difficulties with even getting that to work with Prolog, I would admire it if you could add the complete code when you make suggestions.

Why fail at all? Isn't success better than failure? A shorter version is thus:

main :- writeln('Hello world!').

Also, why even have side effects? Isn't a declarative solution much better:

message('Hello World').

This way, you can also reason about the message within the program itself. This is not possible if it only appears as output on the terminal. You can ask on the toplevel whether there is any message:

?- message(M).
M = 'Hello World'.

File endings and shebang-lines are technicalities that are pretty unrelated to the program itself. A Prolog program that is intended as a shell script will often have an initialization/1 directive somewhere in the file, like:

:- initialization(run).

This is a standardised feature, whereas placing arbitrary queries in the file is not.


DFS and BFS in a graph in C++ Here goes the depth first search:

#include <vector>
#include <iostream>

using namespace std;

typedef vector<vector<int> > graph; // adjacency list, vertexs are integers

void dfs(graph g, int s0) // s0 is the first vertex in the search
{
    int n = g.size();

    if(s0 < 0 || s0 >= n) // invalid initial vertex
        return;

    vector<bool> already_seen(n, false);
    vector<int> to_be_handled {};

    to_be_handled.push_back(s0);

    while(to_be_handled.size() > 0)
    {
        int s = to_be_handled.back();
        to_be_handled.pop_back();

        if(!already_seen[s])
        {
            already_seen[s] = true;

            cout << s << " "; // handle s

            for(int v: g[s]) // add neighbors
                to_be_handled.push_back(v); // (*)
        }
    }
}

And for the breadth first search, just replace the line (*) with this line:

to_be_handled.insert(to_be_handled.begin(), v);

I guess using a queue would be more effective for the BFS but I initially just wanted to implement a DFS so don't mind the second one.

using namespace std is considered bad practice.

to_be_handled.insert(to_be_handled.begin(), v); will badly hurt the performance. As mentioned in comments, BFS benefits from using std::queue, which is specifically designed for this purpose.

Simulating stack with std::vector is also dubious, again from the performance point of view. std::vector guarantees contiguous storage, and the you'd have to face reallocation penalty as it grows. Again, std::stack is specifically designed to avoid such a penalty.

The utility of the function is severely limited by the handler being hardcoded. You may pass the handler as an argument, or better yet restructure the code to make them iterators of Graph, and let the client handle vertices, e.g.:

    Graph g;
    for (auto it: g.bfs_iterator(start)) {
        handle(it);
    }


Copying files into Dropbox folder from command line I made the following code.
Since it is a Batch, any kind of problems can appear.
Please review it and point any problems when copying files into Dropbox!
Its intent is that you will copy a file into an Dropbox folder.
Its syntax is dropbox <files-to-dropbox> <dropbox-folder-to-copy-to>.

@echo off
set dropbox=%userprofile%\dropbox
if exist %dropbox% (if not exist %dropbox%\%2 mkdir %dropbox%\%2) else (echo Get Dropbox for PC before using this!)
if exist %dropbox%\%2 copy %1 %dropbox%\%2
@echo on

It would have been cool to use the dropbox tag as well, right?

Your script will be a lot easier to read if you expand the long statements to multiple lines:

@echo off
set dropbox=%userprofile%\dropbox
if exist %dropbox% (
    if not exist %dropbox%\%2 mkdir %dropbox%\%2
) else (
    echo Get Dropbox for PC before using this!
)
if exist %dropbox%\%2 copy %1 %dropbox%\%2
@echo on

Is that @echo on at the end necessary? Probably not, but I don't have a Windows with me to play around to verify.

The last if statement is pointless to execute if the first if statement was false. It would be better to move this inside the first if statement:

@echo off
set dropbox=%userprofile%\dropbox
if exist %dropbox% (
    if not exist %dropbox%\%2 mkdir %dropbox%\%2
    if exist %dropbox%\%2 copy %1 %dropbox%\%2
) else (
    echo Get Dropbox for PC before using this!
)

Possibly an even better way is to use the equivalent of an early return,
by leaving the main execution path when something is wrong.
You can do this using goto :eof.
After this the else statement can be flattened:

@echo off
set dropbox=%userprofile%\dropbox
if not exist %dropbox% (
    echo Get Dropbox for PC before using this!
    goto :eof
)
if not exist %dropbox%\%2 mkdir %dropbox%\%2
if exist %dropbox%\%2 copy %1 %dropbox%\%2

Rather than using the meaningless symbols %1 and %2 throughout the script,
it would be better to put these values to variables with descriptive names, for example:

@echo off
set source=%1
set target=%2
set dropbox=%userprofile%\dropbox
if exist %dropbox% (
    echo Get Dropbox for PC before using this!
    goto :eof
)
if not exist %dropbox%\%target% mkdir %dropbox%\%target%
if exist %dropbox%\%target% copy %source% %dropbox%\%target%


Iterator to remove duplicate element The code implements custom iterator, which skips duplicate integers. The array expected is to be sorted.

public class GenericIterator<K> implements Iterator<K> {

    private K[] array; // Array will be set here
    private int index;

    GenericIterator(K[] array) {
        this.array = array;
        index = 0;
    }

    /*
     *
     *
     * (non-Javadoc)
     *
     * @see java.util.Iterator#hasNext() /** Returns <tt>true</tt> if this
     * iterator has more elements when traversing the array in the forward
     * direction. (In other words, returns <tt>true</tt> if <tt>next</tt> would
     * return an element rather than throwing an exception.)
     *
     * @return <tt>true</tt> if the array iterator has more elements when
     * traversing the array in the forward direction.
     */

    @Override
    public boolean hasNext() {
        return !(array.length == index);
    }

    /*
     *
     *
     * (non-Javadoc)
     *
     * @see java.util.Iterator#hasNext()
     *
     * /** Returns the next element in the array. This method may be called
     * repeatedly to iterate through the array, or intermixed with calls to
     * <tt>previous</tt> to go back and forth. (Note that alternating calls to
     * <tt>next</tt> and <tt>previous</tt> will return the same element
     * repeatedly.)
     *
     * @return the next element in the array.
     *
     */

    @Override
    public K next() {

        K outputElement = array[index++];
        if (index == 0) {
            index++;
        }

        while (hasNext()) {
            if (array[index] == array[index - 1]) {
                index++;
            } else {
                break;
            }
        }
        return outputElement;

    }
}

Not much to review, but, if I read the Java documentation, you don't put the complete class in it.

Because you didn't put the full class I come to the next thing :

K outputElement = array[index++];
if (index == 0) {
    index++;
}

What the hell are you doing?

index is instantiated at 0, the line before is index++ so index is never 0.

Second point, you missed some errors.
When you are at the last element and you ask next() you will get an OutOfBoundsException.

Rather, it's better to check if your at the last element and if so, just return null.

The OutOfBoundsException is an unchecked exception, so nobody who will use your class will expect and handle this error, and then the problems are coming.

Also add JavaDoc what the Iterator returns when you are at the last element.

Then another point; the constructor has this line :

index = 0;

It's obsolete.  An int default value is always 0.
Because it's not a static variable, it's instantiated when you create the class; so it's 0.

Generics:

With the new description in your question, a few more things come up.
I don't know if you know how to use generics or it's just a fault but:

You expect Integer so I should rename the class name to UniqueIntegerIterator or something like that.
Because public class GenericIterator<K> implements Iterator<K> means that you can insert any type of Object it has a lot more usage then Integer only.
If you want to make it only for Integer, the declaration should change to public class GenericIterator implements Iterator<Integer>.
Of course the K[] should be changed to Integer[]

Expect bad usage of your class.

You state that you expect a sorted integer array in the constructor. You may ask:

  What if this isn't the case?

It's really important to ask yourself this question.
Normally, when you have some requirements, you check for that and throw an AssertionError when the requirements are not met.
In this case and in my own perception, I should sort the array in the constructor so I know the requirement is done.
Also, when I do this I rename the class again to UniqueSortedIntegerIterator so the people who are going to use this class understand they get unique values from a sorted collection, in this case an array.  The Object behind is Integer and it acts like an Iterator.


PHP+JS Code Combining Just wondering if this is okay to do? I have to store some PHP variable values into LocalStorage.

if($res->status == "success"){

            echo '<script>alert("alrighty sparky! lets get you in...");</script>';
            echo '<script>localStorage.setItem("token", JSON.stringify({"token": "'.$res->token.'" }))</script>';
            echo '<script>localStorage.setItem("username", JSON.stringify({"username": "'.$data->username.'" }))</script>';
            echo '<script>localStorage.setItem("id", JSON.stringify({"id": "'.$data->id.'" }))</script>';

            $boarding_url =  base_url() . 'index.php/boarding/teddies';
            redirect($boarding_url);
        }

No, it doesn't work like that because you're not escaping your PHP strings to be a safe JavaScript string. What you're doing is encoding them as JSON but if - for example - $res->token contains \ then it will produce a broken JavaScript string and JSON.stringify() won't fix it.

Correct code must use json_encode() for this purpose.

echo '<script>localStorage.setItem("id", "'.json_encode($data->id).'")</script>';

Also note that you don't need to repeat <script> tag each time:

echo '<script type="text/javascript">';
// Write all lines here, don't forget semicolon at the end of each one
echo 'localStorage.setItem("id", "'.json_encode($data->id).'");';
echo '</script>';


AJAX alternative of using jQuery form UI I was using the jQuery form plugin to process form submission (found here) on my page but now have to switch to using an purely jQuery AJAX based method (without using any the form plugin but I can use jQuery). What would be the best method of achieving this? I'm having difficulty translating it across. What would an ideal solution look like?

// prepare the form when the DOM is ready

$(document).ready(function() {
var options = {
    target:        '#result',
    beforeSubmit:  showRequest,
    success:       showResponse
};

// bind to the form's submit event

$('#booking').submit(function() {
    $(this).ajaxSubmit(options);
    return false;
    });
});

// pre-submit callback

function showRequest(formData, jqForm, options) {
    var queryString = $.param(formData);
    // alert('About to submit: \n\n' + queryString);
}

// post-submit callback

function showResponse(responseText, statusText, xhr, $form)  {
    $('#last-step').fadeOut(300, function() {
    $('#result').html(responseText).fadeIn(300);
    });
}

You can use jQuery's AJAX function for this directly and, you can pass object literals as parameters without using the form elements.

var params = {id : '1234'};
$.ajax({
  type: 'GET',
  url: url, // action attribute from form element
  data: JSON.stringify(params),
  contentType: 'application/json; charset=utf-8',
  dataType: 'json',
  success: function (result) {
      console.log(result);
  },
  error: errorFunc
});

Documentation can be found here.


JavaScript string that holds HTML code I am terrible with writing JavaScript code that outputs HTML. So, here's how a string literal looks like in a project that I'm working on:

tweet = '
'

+ '
'

+ '
'

+ ''
+ ''
+ '	'

+ ''
+ '
'
+ ''
+ '
' + from_user + '
';
})(item.from_user)
+ ''
+ '
' + item.text.twitterify() + '
'
+ '
'
+ '	'

+ '
'

+ '
'

+ '' + item.created_at + ''
+ '
';

How would I go about to improve this?

Make yourself a "template" that you immediately convert into html (using innerHtml or whatever your framework has handy), then use the dom to fix up the few variable bits. You may use ids to locate the tags that need substitution, if you can guarantee they are unique. Otherwise use XPath.


Baby-sitting bash script using DVD drive This is the script I wrote to make an automated POC baby-sitter in 5 lines.

I first found this on a GIF or meme site and thought of making a POC script that does the same.

#How Linux users babysit
while :
do
    eject
    eject -t
done

I don't know aything about bash. So, I want improvements.

You can find the origin and the script at
https://linuxandstuff.wordpress.com/2015/11/21/messing-with-the-dvd-drive-in-command-line/

These days a -T (toggle) argument exists. That allows us to make this a lot neater:

watch -n0 eject -T

You can alter the delay between rocks by changing the -n0 argument.

  watch is used to run any designated command at regular intervals.

http://www.linfo.org/watch.html


API to verify whether a Redis entry exists I come from a PHP object oriented framework background (Laravel, Symfony, Silex...). With that, validation comes in pre-built classes mechanism in a framework that validates for you parameters that you define and validates against. If it fails validation, you don't proceed.

Example from Laravel:

 public function store(Request $request)
    {
        $validator = Validator::make($request->all(), [
            'title' => 'required|unique:posts|max:255',
            'body' => 'required',
        ]);

        if ($validator->fails()) {
            return redirect('post/create')
                        ->withErrors($validator)
                        ->withInput();
        }

        // Store the blog post...
    }

Here, we define that it is required, max 255 for title. It is easy to read.
Symfony is a bit more elaborate but still easy to read due to easily predefined class:

public static function loadValidatorMetadata(ClassMetadata $metadata)
{
    $metadata->addPropertyConstraint('firstName', new Assert\NotBlank());
    $metadata->addPropertyConstraint(
        'firstName',
        new Assert\Length(array("min" => 3))
    );
}

I'm starting to dive into node.js with express.js as base to create an API (I'm aware that Restify and Loopback exist). Currently I have one end point, which I would like to validate a hash token to identify the client, and then inspect that the stored key/value pair matches. The storage is from redis, the following is the code:

/**
 * Check if the hash is in redis
 * @function CheckParam
 * @param {object} req - json object with the data that will be inserted in queries
 * @param cb - callback
 * @callback {object} status code
 */
http.checkParam = function CheckParam(req, cb) {

    if (typeof req.body.data !== 'undefined' && req.body.data && typeof req.query.hash !== 'undefined'
        && req.query.hash && typeof req.body.id !== 'undefined' && req.body.id) {
        if (typeof req.body.data !== 'object' && typeof req.query.hash !== 'string' && typeof req.body.id !== 'number') {
            log.error({part: 'save'}, 'Wrong parameters type');
            cb(403);
        } else {
            redis.get(req.query.hash, function (err, data) {
                if (err) {
                    log.error({part: 'save'}, 'Error from redis \n ' + err);
                    cb(403);
                } else if (!data) {
                    log.error({part: 'save'}, 'Token not found \n ' + req.query.hash);
                    cb(403);
                } else {
                    try {
                        var obj = JSON.parse(data);
                        if (obj.member.id == req.body.id) {
                            cb(200);
                        } else {
                            log.error({part: 'save'}, 'Wrong user id ' + req.body.id);
                            cb(403);
                        }

                    } catch (e) {
                        log.error({part: 'save'}, 'Error from redis \n ' + e);
                        cb(403);
                    }
                }
            });
        }
    } else {
        log.error({part: 'save'}, 'Connection refused');
        cb(403);
    }
};

Is there a more elegant way to validate parameters received from client similar to the above syntax from PHP but in node.js. I reverted to basic "if &&" in order to check 2 simple parameter, but as you can see, it is extensive.

if(typeof req.body.data !== 'undefined' && req.body.data ...

// to

if(req.body.data && req.query.hash && typeof req.body.id)

Assuming that false, 0, null and an empty string has no special meaning in this program, you can simply use loose comparison. Anything falsy (null, undefined, false, 0, NaN, empty string) will just fall off.

function foo(bar){
  if(bar !== 'foo'){
    if(bar !== 'bas'){
      if(bar !== 'bam'){
        // ok, bar is what it is
      } else {
        throw new Error('not bar');
      }
    } else {
      throw new Error('not bar');
    }
  } else {
    throw new Error('not bar');
  }
}

// to

function foo(bar){
  // Expanded for emphasis. You can actually combine
  if(bar === 'foo') throw new Error('not bar');
  if(bar === 'bas') throw new Error('not bar');
  if(bar === 'bam') throw new Error('not bar');

  // ok, bar is what it is
}

Another approach to flattening conditionals is to "return early". Usually, when you have nested if statements, you can invert the condition and the code will look flat. But your mileage may vary, depending on your code.

Now for your errors, HTTP response numbers will cover that for you.

To start, 4xx errors mean the client did something wrong. On the other hand, 5xx errors means the server did something wrong.
A "wrong parameters type" means the client sent the wrong data. That's a 400 (Bad Request) and not a 403 (Forbidden).
A database error or a parse error should be a 500 (Internal Server Error).
If the token does not exist or has the wrong id, that counts as a 401 (Unauthorized). The difference between 401 and 403 is that a 401 can be lifted if you are authenticated (ie. token exists, validation passes etc.) while 403 is more of a permanent unauthorized access.

With that, and removing the logging, this should be all that's left. Feel free to add back the brackets. I usually omit them if I deal with one-liner bodies.

http.checkParam = function CheckParam(req, cb) {
  if(!req.body.data || !req.query.hash || ! req.body.id) cb(400);
  else redis.get(req.query.hash, function(err, data) {
    if(err) cb(500);
    else if(!data) cb(401);
    else try {
      var obj = JSON.parse(data);
      if (obj.member.id !== req.body.id) cb(401);
      else cb(200);
    } catch(error){
      cb(500);
    }
  });
};


Parse date format in Pandas using Python I have a column in a Pandas Dataframe containing birth dates in object/string format:

0    16MAR39
1    21JAN56
2    18NOV51
3    05MAR64
4    05JUN48

I want to convert the to date formatting for processing. I have used

#Convert String to Datetime type
data['BIRTH'] = pd.to_datetime(data['BIRTH'])

but the result is ...

0   2039-03-16
1   2056-01-21
2   2051-11-18
3   2064-03-05
4   2048-06-05
Name: BIRTH, dtype: datetime64[ns]

Clearly the dates have the wrong century prefix ("20" instead of "19")

I handled this using ...

data['BIRTH'] = np.where(data['BIRTH'].dt.year > 2000, data['BIRTH'] - pd.offsets.DateOffset(years=100), data['BIRTH'])

Result

0       1939-03-16
1       1956-01-21
2       1951-11-18
3       1964-03-05
4       1948-06-05

Name: BIRTH, Length: 10302, dtype: datetime64[ns]

I am wondering:

if there is a way to process the data that will get it right first time?
If there is a better way to process the data after the incorrect conversion.

I'm an amateur coder and as far as I understand things Pandas is optimised for processing efficiency. So I wanted to use the Pandas datatime module for that reason. But is it better to consider Numpy's or Pandas' datetime module here? I know this dataset is small but I am trying to improve my skills so that when I am working on larger datasets I know what  to consider.

Source data

This post on Stack Overflow explains why you are getting the wrong years.

https://stackoverflow.com/questions/37766353/pandas-to-datetime-parsing-wrong-year

Based on your code all of the two digit years in your data set will be converted to 19XX years. The only problem I can see is that if your data set includes dates across both centuries ( 19XX and 20XX) you'll end up forcing anything that should be 20XX to be 19XX the way you have it written. If your data set has dates in both centuries I'd recommend preprocessing your date strings to make them unambiguous (change 16MAR39 to 16MAR1939). This will require additional information from another tag in your data set if you've got it.

To your specific questions:

Since the data is ambiguous there isn't a way to get it right the
first time.  If you preprocess the data then it should work as you
want with the single pd.to_datetime command.
Processing the data on the front end to resolve the ambiguity (based on other information in your set) is probably a better solution than assuming you need to offset every date after 2000. For example, 1MAR05 will be read as 2005, then your code will offset by 100 years and you'll get 1905 when maybe it should have actually been 2005.

Amateur coder here learning too, but I don't think there is a built in function to make your data unambiguous or force a specific century prefix.


Read daily files and concatenate them Edit - here is my modified code: http://jsfiddle.net/#&togetherjs=GzytydCsRh

Can someone take a look and give me some feedback? It seems a bit long still but that is the first time I used functions.

I am reading a bunch of CSV files and using glob to concatenate them all together into separate dataframes. I eventually join them together and basically create a single large file which I use to connect to a dashboard. I am not too familiar with Python but I used Pandas and sklearn often.

As you can see, I am basically just reading the last 60 (or more) days worth of data (last 60 files) and creating a dataframe for each. This works, but I am wondering if there is a more Pythonic/better way? I watched a video on pydata (about not being restricted by PEP 8 and making sure your code is Pythonic) which was interesting.

(FYI - the reason I need to read 60 days worth of files is because customers can fill out a survey from a call which happened a long time ago. The customer fills out a survey today about a call that happened in July. I need to know about that call (how long it lasted, what the topic was, etc).

import pandas as pd
import numpy as np
from pandas import *
import datetime as dt
import os
from glob import glob
os.chdir(r'C:\\Users\Documents\FTP\\')
loc = r'C:\\Users\Documents\\'
rosterloc = r'\\mand\\'
splitsname = r'Splits.csv'
fcrname = r'global_disp_'
npsname = r'survey_'
ahtname = r'callbycall_'
rostername = 'Daily_Roster.csv'
vasname = r'vas_report_'
ext ='.csv'
startdate = dt.date.today() - Timedelta('60 day')
enddate = dt.date.today()
daterange = Timestamp(enddate) - Timestamp(startdate)
daterange = (daterange / np.timedelta64(1, 'D')).astype(int)

data = []
frames = []
calls = []
bracket = []
try:
    for date_range in (Timestamp(startdate) + dt.timedelta(n) for n in range(daterange)):
        aht = pd.read_csv(ahtname+date_range.strftime('%Y_%m_%d')+ext)
        calls.append(aht)
except IOError:
        print('File does not exist:', ahtname+date_range.strftime('%Y_%m_%d')+ext)
aht = pd.concat(calls)
print('AHT Done')
try:
    for date_range in (Timestamp(startdate) + dt.timedelta(n) for n in range(daterange)):
        fcr = pd.read_csv(fcrname+date_range.strftime('%m_%d_%Y')+ext, parse_dates = ['call_time'])
        data.append(fcr)
except IOError:
        print('File does not exist:', fcrname+date_range.strftime('%m_%d_%Y')+ext)
fcr = pd.concat(data)
print('FCR Done')
try:
    for date_range in (Timestamp(enddate) - dt.timedelta(n) for n in range(3)):
        nps = pd.read_csv(npsname+date_range.strftime('%m_%d_%Y')+ext, parse_dates = ['call_date','date_completed'])
        frames.append(nps)
except IOError:
        print('File does not exist:', npsname+date_range.strftime('%m_%d_%Y')+ext)
nps = pd.concat(frames)
print('NPS Done')
try:
    for date_range in (Timestamp(startdate) + dt.timedelta(n) for n in range(daterange)):
        vas = pd.read_csv(vasname+date_range.strftime('%m_%d_%Y')+ext, parse_dates = ['Call_date'])
        bracket.append(vas)
except IOError:
        print('File does not exist:', vasname+date_range.strftime('%m_%d_%Y')+ext)
vas = pd.concat(bracket)
print('VAS Done')
roster = pd.read_csv(loc+rostername)
print('Roster Done')
splits = pd.read_csv(loc+splitsname)
print('Splits Done')

Use a class, or at least some functions, to make your code more readable and understandable

Very first reaction looking at your code is ....blech. I don't want to read that giant blob.
Why not make a class to bundle a bunch of functions together, such as a function readAndConcatAHT? Actually, many of these for loops are doing the exact same thing for slightly differently named files. Why not do something like a function that takes in a filename and then runs a for loop like so:

def readAndConcatFile(filename, daterange):
    try:
        for date_range in (Timestamp(startdate) + dt.timedelta(n) for n in range(daterange)):
            fcr = pd.read_csv(filename+date_range.strftime('%m_%d_%Y')+ext, parse_dates = ['call_time'])
            data.append(fcr)
    except IOError:
        print('File does not exist:', fcrname+date_range.strftime('%m_%d_%Y')+ext)

This would really clear out your code even if you elected not to write any other functions or a call. I think it's also fairer to your reader and yourself to respect DRY and not make readers check for themselves when you are doing something absolutely repetitive with slightly different function names.

I'll put an extra point to say I think a class would be nice because in your init or some processing function you could string together a bunch of calls to readAndConcatFile to standardize your read/write process for these CSV files. This will, again, make your code more extensible and more readable.

Avoid redundant import statements and stick with standards

Almost everyone uses import pandas as pd. I wouldn't recomend doing it any other way, and it's never a good idea to do a whole scale import *
Don't import glob unless you are actually using it. Where do you use glob after importing it?

Use special features only when you need them

Do you actually need raw strings? I don't see you using your strings in any way that would seem to require them.
Similarly, why use os.chdir when it could be smarter to specify filenames as absolute file names? Here you're again using an option you don't really need and that could have future unintended side effects.

Use more defined constants

It's not a good idea to hard code Timedelta(60 day) like so. You should separately specify DAY_RANGE = 60 as a constant and then use that wherever you'd use 60. That way you can easily change the day range. Alternately, you could make the day range an input parameter to your script so that non-programmer users can also call this script for their desired look-back period.
Similarly, you can save your desired date formats as strings to be treated as constants at the top of your file:
format1 = "'%m_%d_%Y'" and format2 = "'%Y_%m_%d'" Again this makes it easier to see what's going on and also makes it easier to make changes in the future. You can change just one string at the top of your file to change all related formatting, rather than having to change each string. This won't make any given line of code shorter, but it will make it better.

More sophisticated error handling

Error handling is not something I do enough of myself, but I wonder if you can do better here. I'm going to assume that errors in ahtname are related, for example, to errors in fcrname. If that's the case, once you establish that a date range is missing for one kind of file, why not delete that daterange for all further queries in future loops? You could do so easily by simply deleting that member of daterange that causes the IOError. Then you wouldn't get repetitive error messages that are really all telling you the same thing.

What I liked

It's good practice to use generator expressions where you can, so I liked seeing code like for date_range in (Timestamp(startdate) + dt.timedelta(n) for n in range(daterange)):.


Faster brute force algorithm I have this brute force code, where you input a password and runs through combinations of numbers, lowercase and uppercase letters, and special characters until it match the password given.

The problem with it, is that it took about 2 days just to crack the password "password".

What is a way that I can speed up this process and get the passwords faster? Is there a way I can implement multi-threading to speed up the process?

All I'm really asking is, is there a better way for the program to run through all of the characters and match the password faster?

//This program is used for a school project, where I have to create a list of commonly used passwords and run them through the program and see how long it take for the program to crack them. *I would like to get the best result possible and speed up the process*

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.InteropServices.ComTypes;
using System.Text;

  namespace Bruteforce
 {
   class Program
 {
   //define likely password characters
    static char[] Match =            {'0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f','g','h','i','j' ,'k','l','m','n','o','p',
                        'q','r','s','t','u','v','w','x','y','z','A','B','C','D','E','F','G','H','I','J','C','L','M','N','O','P',
                        'Q','R','S','T','U','V','X','Y','Z','!','?',' ','*','-','+'};
//your password
static string FindPassword;
static int Combi;
static string space;
static int Characters;

static void Main(string[] args)
{
    Console.Title = "Brute Force";

    Console.ForegroundColor = ConsoleColor.Green;
    Console.WriteLine("ALL FILES LOADED...");
    Console.ForegroundColor = ConsoleColor.White;

    space = " ";
    int Count;

    //user types in a password

    Console.WriteLine("Welcome to BRUTE FORCE");
    Console.WriteLine("This program was crated by Nick");
    Console.WriteLine(space);
    Console.WriteLine("Enter your Password:");

    //initialize your password
    FindPassword = (Console.ReadLine());
    Characters = FindPassword.Length;
    Console.Clear();

    DateTime today = DateTime.Now;
    today.ToString("yyyy-MM-dd_HH:mm:ss");
    Console.WriteLine(space);
    Console.WriteLine("START:\t{0}", today);

    for (Count = 0; Count <= 15; Count++)
    {
        Recurse(Count, 0, "");
    }
}

static void Recurse(int Lenght, int Position, string BaseString)
{
    int Count = 0;

    for (Count = 0; Count < Match.Length ; Count++)
    {
        Combi++;
        if (Position < Lenght - 1)
        {
            Recurse(Lenght, Position + 1, BaseString + Match[Count]);
        }
        if (BaseString + Match[Count] == FindPassword)
        {
            Console.WriteLine();
            Console.ForegroundColor = ConsoleColor.Blue;
            Console.WriteLine("Your password is: " + FindPassword);
            Console.WriteLine("Your password is: " + Characters + " character(s) long");
            Console.ForegroundColor = ConsoleColor.White;
            DateTime today = DateTime.Now;
            today.ToString("yyyy-MM-dd_HH:mm:ss");
            Console.WriteLine(space);
            Console.WriteLine("END:\t{0}\nCombi:\t{1}", today, Combi);
            Console.ReadLine();
            Environment.Exit(0);
        }
    }
  }
 }
 }

Algorithmically speaking, there is no better way to find a matching password without considering characteristics of the password itself.  For instance, if you take into account the length, you could restrict the generated passwords to be only of that length.  If you consider the actual characters, then well, you already have the password, so you're done.  But neither of these seem to be true to the exercise, do they?

But what about parallelization?  After all, this task is trivial to parallelize.  Surely if we harness the processing power of all our cores, we will be able to chew through those passwords in lightning speed, right?

Ok, let's talk about that.  Suppose you have a machine with 24 cores.  If we parellelize, should get a speedup of 24, right?  Well, not quite, because there's always some overhead when you parallelize something, due to things like task coordination and so on.  So the actual speedup will be something less than 24.  But, for the sake of argument, let's assume we have an ideal system and we actually do see a speedup of 24.

Now, you said it took 2 days to find the password "password", right?  So with 24 cores, it now takes 2 hours.  "password" is 8 charaters.  So let's just say that we can find any 8 character password in 2 hours.  That's pretty good compared to 2 days!

But what about 9 character passwords.  If I counted correctly, there are 68 characters you use to generate passwords.  So the set of all 9 character passwords will be 68 times larger than the set of all 8 character passwords.  So generating all 9 character passwords will take 68*2 hours = 136 hours = 5 days + 16 hours.  Ouch!

So what now, purchase more machines?

By the way, the maximum password length you examine is 15.  There are 68^15 passwords of that length.  8^15 is 3,073,503,348,387,795,563,479,826,432.  To examine this number of passwords will take 6,722,988,818,432 times longer than it takes to examine all 8 character passwords.  And yes, that's 6.7 Trillion.  It would about 1.5 Billion years to examine all those passwords on our 24-core machine.  I don't know how old you are, but suffice it to say that if you plan on obtaining the results before you expire, you're going to need a lot more machines.  If you do plan on doing this, please let me know what server provider you'll be using ... I'd love to invest in them!

So, the bottom line is, the only way to brute force something is through brute force.  Parallelization can help, but only if the problem is tractable to begin with.  The power of exponentiation will win every time.


Division without multiplication, addition or division In context of preparing for a coding interview, I gave the division problem a go using Java. This is what I came up with. It's a little bit different to other solutions I've come across the interweb.

  Implement integer division without using multiplication or repeated subtraction (i.e. to divide n by d, you man not repeatedly subtract off d from n).

public static int division(int a, int b) {

    // For the basic cases
    if (b == 1) { return  a; }
    if (a == 1) { return  b; }
    if (a == 0 || b== 0) { return  0; }
    if (b > a)  { return  0; }
    if (a == b) { return  1; }

    int result = 0;
    int size = size(a) - 1;
    for (int i = size(a) - size(b); i >= 0; i--) {
        if((a - (b << i) >= 0)) {
            a = a - (b << i);
            result = result + (1 << i);
        }
    }

    return result;
}

public static int size(int a) {
    // The index of the most left 1 in the binary structure
    int temp = new Integer(a);
    int depth = 0;
    while (a > 0) {
        a >>= 1;
        depth++;
    }
    return depth;
}

If you run

Random rand = new Random();
for (int i = 0; i < 100; i++) {
    int a = rand.nextInt();
    int b = rand.nextInt();
    System.out.println(a + " / " + b + " == " + division(a, b) + " == " + a / b);
}

You'll see very clearly that the code is buggy. Investigate this yourself, and then come back to this post.

if (a == 1) { return  b; }

Is simply wrong.

if (a == 0 || b== 0) { return  0; }

should throw a java.lang.ArithmeticException: / by zero in the latter case. It should do so before checking a.

if (b > a)  { return 0; }

Covers the case of a == 0, as long as you ignore negatives - so remove the a == 0 check.

You don't even need these checks, though, so remove them.

You don't use

int size = size(a) - 1;

and it's also really bad style to have both an integer and a function of the same name anyway.

a = a - (b << i);
result = result + (1 << i);

should use -= and +=.

a - (b << i) >= 0

should be

a >= b << i

One then has

public static int division(int a, int b) {
    if (b == 0) {
        throw new java.lang.ArithmeticException("/ by zero");
    }

    int result = 0;
    for (int i = size(a) - size(b); i >= 0; i--) {
        if (a >= b << i) {
            a -= b << i;
            result += 1 << i;
        }
    }

    return result;
}

One should deal with negative numbers. This looks as simple as

boolean negate = false;
if (a < 0) {
    a = -a;
    negate ^= true;
}
if (b < 0) {
    b = -b;
    negate ^= true;
}

...

if (negate) {
    return -result;
} else {
    return result;
}

However, this fails for INT_MIN since -INT_MIN == INT_MIN!

What to do? It seems like one could flip into the negatives instead, and flip all of the comparisons. I don't think this would work, though, since

a = INT_MIN
b = INT_MIN + 1

would result in i = 1, resulting in running b << 1, which overflows. This doesn't just happen with INT_MIN either.

The simplest thing, then, might be to flip to positive after casting to long. One can also simplify things by removing size, then, in favour of some fiddling with b like:

int i = 1;
while (b < a) {
    b <<= 1;
    i <<= 1;
}

long result = 0;
for (; i > 0; i >>= 1, b >>= 1) {
    if (a >= b) {
        a -= b;
        result += i;
    }
}

All in all, one gets the unfortunately long

public static int division(int left, int right) {
    // Prevent overflow on negation
    long a = left;
    long b = right;

    if (b == 0) {
        throw new java.lang.ArithmeticException("/ by zero");
    }

    // Normalize to positive longs.
    boolean negate = false;
    if (a < 0) {
        a = -a;
        negate ^= true;
    }
    if (b < 0) {
        b = -b;
        negate ^= true;
    }

    int i = 1;
    while (b < a) {
        b <<= 1;
        i <<= 1;
    }

    long result = 0;
    for (; i > 0; i >>= 1, b >>= 1) {
        if (a >= b) {
            a -= b;
            result += i;
        }
    }

    if (negate) {
        // Should not overflow
        return (int)-result;
    } else {
        // Truncation might overflow if
        // result is -(long)Integer.MIN_VALUE,
        // but this is correct
        return (int)result;
    }
}


Why does my function think that "uiouioiu"is a plaindrome? I was reading Introduction to Computation and Programming using Python and there in an exercise, we had to write a function which would tell us if a given word is a palindrome or not, recursively.

This is the code which I wrote:

word = input()

def isPalindrome(x):
    if len(x) > 1:
        if x[0] == x[-1] and isPalindrome(x[1:-1]):
            return('The word is Plaindrome.')
        else:
            return('It is not.')
    else:
        return(True) #As this is a base case.

print(isPalindrome(word))

This seems to work for all the words I pass into it except uiouioiu.
isPalindrome(uiouioiu) returns The word is Palindrome.

Could someone explain what is happening?

EDIT 1: If I modify the code to:

word = input()

def isPalindrome(x):
    if len(x) > 1:
        if x[0] == x[-1] and isPalindrome(x[1:-1]):
            return(True)
        else:
            return(False)
    else:
        return(True) #As this is a base case.

print(isPalindrome(word))

isPalindrome(uiouioiu) returns False as expected. Why is this happening?

The reason it doesn't work is that "It is not" is converted into True, so when you analyze
"uiouioiu" the process eventually has to check the middle part "ui", the problem is that the answer "It is not" is then interpreted as True.

The same behaviour should also be obtained when testing the word "abca"