Returning a NumPy array (or list) of strings of words repeated N times I have a list of \$n\$ words, and a corresponding \$m \space x \space n\$ frequency matrix (as a NumPy array). I would like to return a list/array of strings of length \$m\$ where the \$m\$th string is comprised of each word repeated according to the frequencies in the \$m\$th row of the frequency matrix. I have managed to achieve the desired result (help from here), but the code is not particularly easy to understand at a glance. Is there a cleaner and more efficient way to perform the following operation?

import numpy as np
x = ['yugoslavia', 'zealand', 'zimbabwe', 'zip', 'zone']
y = np.array([[2,1,0,0,5], [0,0,1,3,0]])
z = np.apply_along_axis(lambda b: ' '.join([ item for sublist in [[x[i]]*b[i] for i in range(len(x))] for item in sublist]),1,y)

>>> z
array(['yugoslavia yugoslavia zealand zone zone zone zone zone',
   'zimbabwe zip zip zip'],
  dtype='<U54')

I am looking for solutions compatible with Python 3.5.

It would seem you're doing it the right way. One thing though: you might want to replace the following piece of code:

[[x[i]]*b[i] for i in range(len(x))]

A few points as to how you could improve this:

I suggest you use zip to iterate over two arrays simultaneously.
Also, prefer using () over [], since it creates a generator expression, rather than a list.
A similar argument holds with the construct join([ ... ]). Simply use join( ... ) instead, which would avoid creating the list in memory.
Better variable names will also help with clarity.

([s] * count for s, count in zip(strings, counts))

Finally, formatting can make loads of difference:

import numpy as np

strings = ['yugoslavia', 'zealand', 'zimbabwe', 'zip', 'zone']
counts_array = np.array([[2,1,0,0,5], [0,0,1,3,0]])
result = np.apply_along_axis(
    lambda counts: ' '.join(item for sublist in
                                ([s] * count for s, count in zip(strings, counts))
                            for item in sublist),
    1, counts_array)

An equally ugly alternative might involve using two join statements:

result = np.apply_along_axis(
    lambda counts: ' '.join(filter(None,
                  (' '.join([s] * count) for (s, count) in zip(strings, counts)))),
    1, counts_array)

Note how I've had to use filter, as per this question, in order to remove the extra spaces emanating from the empty strings.


Haskell requires more memory than Python when I read map from file. Why? I have this simple code in Python:

input = open("baseforms.txt","r",encoding='utf8')
S = {}
for i in input:
    words = i.split()
    S.update( {j:words[0] for j in words} )
print(S.get("sometext","not found"))
print(len(S))

It requires 300MB for work. "baseforms.txt" size is 123M.

I've written the same code in Haskell:

{-# LANGUAGE OverloadedStrings #-}
import qualified Data.Map as M
import qualified Data.ByteString.Lazy.Char8 as B
import Data.Text.Lazy.Encoding(decodeUtf8)
import qualified Data.Text.Lazy as T
import qualified Data.Text.Lazy.IO as I
import Control.Monad(liftM)

main = do
    text <- B.readFile "baseforms.txt"
    let m = (M.fromList . (concatMap (parseLine.decodeUtf8))) (B.lines text)
    print (M.lookup "sometext" m)
    print (M.size m)
    where
        parseLine line = let base:forms = T.words line in [(f,base)| f<-forms]

It requires 544 MB and it's slower than Python version. Why? Is it possible to optimise Haskell version?

Data file can be downloaded here

I've tried non-lazy version of Data.Text and Data.Text.IO - memory usage is near 650MB

I've also tried hashtables, but  memory usage growth up to 870 MB

I can't seem to download your data set. Could you re-upload a smaller dataset (say, ~5mb? and with your testing statistics for it) to Pastebin or somewhere?

The only obvious performance difference with your Python version is that Data.Map is based on a binary tree, whereas Python's dict is a hashtable. This makes fromList an O(n * log n) operation in Haskell (unless you luck into the fact that your list of (form, base) pairs happen to all be distinct and sorted in ascending order) but constructing the Python dict is O(n). This should make a significant time difference, but not necessarily space, so at this point I'm out of ideas. Make sure you're compiling with -O2 I guess.

(There also seems to be a functional difference between your two versions if my understanding of Python hasn't rusted off after years of neglect. {j:words[0] for j in words} will pair words[0] with itself, so maybe you want parseLine to be defined as let forms@(base:_) = ....)

On issues of style, see the below rewrite. Note particularly the reordering of definitions so as to eliminate the where-clause. In your version it was easy to miss due to being inline with the do-block, you can do as I did, or try decreasing the indent.

{-# LANGUAGE OverloadedStrings #-}

import qualified Data.Map             as Map
import qualified Data.ByteString.Lazy as ByteString
import qualified Data.Text.Lazy       as Text
import           Data.Text.Lazy.Encoding (decodeUtf8)

main :: IO ()
main = do
    text <- fmap decodeUtf8 $ ByteString.readFile "baseforms.txt"
    let parseLine line = let (base:forms) = Text.words line
                         in  zip forms (repeat base)
        m = Map.fromList . concatMap parseLine . Text.lines $ text
    print $ Map.lookup "sometext" m
    print $ Map.size m


Naming a template print method properly Some might not think it's worth asking but I consider good function names quite important. I am not sure what should be the overridable function name in what I think is template design pattern. I am printing a report which has 3 types. There are lots of if-else which is confusing. I replace the long function with the polymorphic class hierarchy with base class below. I have than derived a class from each report type which overrides the PrintCustom() function but this doesn't strike me as a good name.

class CPrintReport
{
protected:
    CPrintReport(void);
public:
    virtual ~CPrintReport(void);

    // template function?
    void PrintReport();

protected:
    void PrintCommon();
    virtual void PrintCustom() = 0;
    virtual void PrintChart();

protected:
    CStringArray pstrData;

    CGraphCtrl * m_GraphCtrl;

};

PrintReport is the main function which is as following:

void CPrintReport::PrintReport()
{
    // prints thte common data
    PrintCommon();

    // prints specialized (verification,identification or regular type)
    PrintCustom();

    PrintChart();

}

Does this implement the template design pattern where PrintReport() is the template function?
As you can see from report, the first section prints common information, and the last section prints a chart, but the middle section varies based on what type of report it is. If it was procedural programming I would be calling this PrintVerificationData()and PrintIdentifcationData() depending on which type of report is being printed. How do we name such methods in the template design pattern so it makes better sense?
It turns out PrintChart is going to be different as well for each type. However, it is the same for 2 types (verification and identification) and different for regular print. Does this mean PrintChartVerifyIdentify() (being lazy but making a point) should reside in the base class so it can be shared, or should I use a visitor pattern or something?

I think you're right to be concerned with the names you're using.

I'd start with the name of the base class. CPrintReport strikes me as a rather poor name. First of all, it sounds like a verb phrase (Print Report). Most classes should be nouns. Second, you typically want the name to reflect the fact that it's an abstract base class. Finally, tacking a C onto the beginning of a class name is, frankly, a pretty lousy idea.

I'd probably start with a report namespace, and name this base:

namespace report {
    class Base {
        // ...
    };
}

The other reports would derive from it, so you might have something like:

namespace report {
    class Verification : public Base {
        // ...
    };

    class Identification : public Base {
        // ..
    };
}

Then when you want to print one, your code would be something like:

report::Verification verification_report;
verification_report.print(printer);

As far as the names of the individual functions go, I think the problem is fairly simple: right now, your names basically reflect the mechanics of generating the reports. I'd prefer to see names that reflect the sections of the report they generate instead.

For example, your PrintCommon, PrintCustom and PrintChart might become: PrintIntroduction, PrintDetails, and PrintSummary respectively. Of course, I don't know for sure how your reports are structured, so there might be other names that make more sense, but the basic idea still applies--to the extent possible, base the names on their effect in the final product rather than on the details of how you're implementing the code.


Speeding up VBA Code that Sets Pivot Table Filters I've written out some VBA code that sets various filters onto pivottables. The problem I am experiencing is that it is extremely slow and bogs down my system heavily.

I've tried implementing various things to speed up the code, but nothing really has that much of an effect.

The reason I am doing it this way is the filters need to be dynamic and I am using multiple data sources, so just using a single slicer doesn't work in my case.

Here is my code:

 Private Sub Worksheet_Calculate()

    Application.ScreenUpdating = False
    Application.EnableEvents = False
    Application.Calculation = xlCalculationManual
    Application.DisplayStatusBar = False
    ActiveSheet.DisplayPageBreaks = False

    Dim DivRef, RegRef, DistRef, ZoneRef As String
    DivRef = Sheet5.Range("AH6").Value
    RegRef = Sheet5.Range("AH7").Value
    DistRef = Sheet5.Range("AH8").Value
    ZoneRef = Sheet5.Range("AN4").Value

With Sheet5.PivotTables("PivotTable21")
    .ManualUpdate = True
    .PivotFields("Division2").CurrentPage = DivRef
    .PivotFields("Region2").CurrentPage = RegRef
    .PivotFields("District2").CurrentPage = DistRef
    .ManualUpdate = False
End With

With Sheet5.PivotTables("PivotTable9")
    .ManualUpdate = True
    .PivotFields("Division2").CurrentPage = DivRef
    .PivotFields("Region2").CurrentPage = RegRef
    .PivotFields("District2").CurrentPage = DistRef
    .ManualUpdate = False
  End With

     Application.EnableEvents = True
     Application.ScreenUpdating = True
     Application.Calculation = xlCalculationAutomatic

    End Sub

In reality, there are a number of pivottables that are being filtered, but for the sake of this post, I am only posting PivotTable21 and PivotTable9's filters, though it is all more or less the same code.

I've been told that part of the problem might be that I am using calculated fields in my pivottables, and that may bog down some of the code as well.

Does anyone have any tips for speeding this code up?

I find that when you set PivotTable.ManualUpdate = True, Excel seems to often set that back to False for many of the changes you can make to PivotTables.  It does depend on the change, but I think changing PivotFields is one of those things that triggers this.

So you could try going to every PivotField change and resetting ManualUpdate to True after it.  Such as this:

With Sheet5.PivotTables("PivotTable21")
    .ManualUpdate = True
    .PivotFields("Division2").CurrentPage = DivRef
    .ManualUpdate = True
    .PivotFields("Region2").CurrentPage = RegRef
    .ManualUpdate = True
    .PivotFields("District2").CurrentPage = DistRef
    .ManualUpdate = False
End With

I know it seems silly, but I've seen this work.


Method to count interior nodes of a binary tree I'm not a programmer, but am playing around with a binary tree Class in Python and would like help with a recursive method to count interior nodes of a given tree.

The code below seems to work, but is this solution the clearest, in terms of logic, for counting interior nodes? I'm keen that the code is the cleanest and most logical solution, whilst remaining recursive.

The qualifier for being an interior node, as far as I'm aware here is, that an interior node should have a parent, and at least one child node.

def count_interior_nodes(self, root):
    """Return number of internal nodes in tree"""
    if(root.parent == None):
        # Root node so don't count, but recurse tree
        return self.count_interior_nodes(root.left) + self.count_interior_nodes(root.right) + 0
    elif (root.left != None or root.right != None):
        # Has parent, but isn't leaf node, so count and recurse
        return self.count_interior_nodes(root.left) + self.count_interior_nodes(root.right) + 1
    else:
        # Leaf node, so return zero
        return 0

So after our little discussion, I hope you see that you're missing some key cases here, when the root given is None.

Some things that should be pointed out, when comparing to None, you should be checking if it is or is not None.  Don't use (in)equality here, it is just not correct.

Assuming this is a method on your tree node objects, I'd rewrite this to make use of self.  That way we don't even need to worry about the case when self is None, that just shouldn't happen.

def count_interior_nodes(self):
    count = 0
    hasLeft, hasRight = self.left is not None, self.right is not None
    if hasLeft:
        count += self.left.count_interior_nodes()
    if hasRight:
        count += self.right.count_interior_nodes()
    if (hasLeft or hasRight) and self.parent is not None:
        count += 1
    return count


Setting a textbox value to zero if null Is this the correct way to set value of textbox to zero when it's null? Is there any way I can improve the calculation that I am doing?

private void TB_PAID_CASH_TextChanged(object sender, EventArgs e)
{
    try
    {
        if (!string.IsNullOrEmpty(TB_TOTAL_INV.Text) && !string.IsNullOrEmpty(TB_PAID_CASH.Text) && string.IsNullOrEmpty(TB_Discount.Text))
        {
            int Discount;
            int.TryParse(TB_Discount.Text, out Discount);
            TB_REMAINDER.Text = (Convert.ToInt32(TB_PAID_CASH.Text) - Convert.ToInt32(TB_TOTAL_INV.Text) - Discount).ToString();
        }
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.Message, "Exception", MessageBoxButtons.OK, MessageBoxIcon.Error);
    }
}

You are mixing various things here. For some values you use the try/catch and for others the TryParse. I think you should stick to one of them.

What I also don't understand is why you are checking whether this is true string.IsNullOrEmpty(TB_Discount.Text) and then try to parse it anyway int.TryParse(TB_Discount.Text)?

Here is what you can do:

Create a property for each value:

public int TotalInv
{
    get
    {
        int totalInv;
        return int.TryParse(TB_TOTAL_INV.Text, out totalInv) ? totalInv : 0;
    }
}

public int PaidCash
{
    get
    {
        int paidCash;
        return int.TryParse(TB_PAID_CASH.Text, out paidCash) ? paidCash : 0;
    }
}

public int Discount
{
    get
    {
        int discount;
        return int.TryParse(TB_Discount.Text, out discount) ? discount : 0;
    }
}

For the reminder too:

public int Reminder
{
    get
    {
        return PaidCash - TotalInv - Discount;
    }
}

Then when the text changes you can update the text box with only one line:

private void TB_PAID_CASH_TextChanged(object sender, EventArgs e)
{
    if(PaidCash < TotalInv)
    {
         // todo: show message...
    }
    TB_REMAINDER.Text = Reminder.ToString();
}


Compare / Join two object collections LINQ I have two csv files that I need to compare and merge together. There is a 'big' list with many records and columns of data and a 'small' list of the desired records and some additional data. The ultimate goal is to output the records on the small list joined with the data in the big list.

The below code seems to make the correct selects, but I am unsure if I should be looking at using JOIN instead?

I like the cleanness of this solution in how it handles the CSV files in such few lines of code - let ... some.Split(',') - and haven't come up with as clean of a way in using JOIN.

string[] small = System.IO.File.ReadAllLines(@"I:\\" + Environment.UserName + "\\smallfile.csv");

string[] big = System.IO.File.ReadAllLines(@"J:\\DROPFOLDERS\\bigfile.csv");

IEnumerable<MBSGroup> queryMBSObjects =
from smallLine in small
let splitSmallLine = smallLine.Split(',')
from bigLine in big
let splitBigLine = bigLine.Split(',')
where ((splitSmallLine[0] == splitBigLine[0] && splitSmallLine[2] == splitBigLine[1] && splitSmallLine[0] != "" && splitSmallLine[2] != "" && splitBigLine[1] != "") ||
       (splitSmallLine[0] == splitBigLine[0] && splitSmallLine[2] == splitBigLine[5] && splitSmallLine[0] != "" && splitSmallLine[2] != "" && splitBigLine[5] != ""))
select new MBSGroup()
{
    write data to merge object
};
List<MBSGroup> MBSlists = queryMBSObjects.ToList();

This code seems to suffer from poor readability and maintainability. What does splitSmallLine[2] mean, and why do I care if it matches splitBigLine[1] (ponders the next developer to run into this code)? It's very easy to make mistakes in this code, so I think bringing some clarity into what you are actually selecting/comparing against will help.

Since I don't actually know what your files look like, or what the fields represent, let's just start here:

string[] small = new string[] {
    "Joe,Smith,22",
    "Sally,Jones,32"
};

string[] big = new string[] {
    "Smith,22,Joe",
    "Jones,34,Sally",
    "Doe,15,Jane"
};

We have two "files" that store people, but the fields aren't in the same place. The first thing you could do is bring meaning to your parsed rows, like so:

var smallFilePeople = from personLine in small
                      let personFields = personLine.Split(',')
                      select new
                      {
                          FirstName = personFields[0],
                          LastName = personFields[1],
                          Age = personFields[2]
                      };

var bigFilePeople = from personLine in big
                    let personFields = personLine.Split(',')
                    select new
                    {
                        FirstName = personFields[2],
                        LastName = personFields[0],
                        Age = personFields[1]
                    };

Now I know that I'm selecting people, and I have a single place to show what I'm actually pulling out of each indexed location. This makes writing a join query based on this very simple and easy to understand:

var joined = from smallFilePerson in smallFilePeople
             join bigFilePerson in bigFilePeople
                on new { smallFilePerson.FirstName, smallFilePerson.LastName } equals
                   new { bigFilePerson.FirstName, bigFilePerson.LastName }
             select new
             {
                 FirstName = smallFilePerson.FirstName,
                 LastName = smallFilePerson.LastName,
                 AgeInSmallFile = smallFilePerson.Age,
                 AgeInBigFile = bigFilePerson.Age
             };

This example is a bit more contrived than yours, but I'm sure you can see where this may help. I would also take this a step further and split out the query/parsing logic/joining into distinct functions. It's a good separation of concerns, and you may end up reusing the parsing for each file in other new ways.


c# oop, is that correct? I have kinda problem with oop in c#, I need to do a task to school but I completely new in oop. At this moment I have no idea how I can even start this, I mean I don't know how to make that program counts people... like e.g. click spacebar and it will make counter of people increase. Maybe someone could help me to solve this out. The task is:

Create a class(called Counter) that will check how many people entered the building(like the gate in the entrance counts how many people entered or how many is inside already). The class should contain a methods that will singly increase amount of people or singly decreases amount of them, the third method should show how many is inside already. Last but not least, the constructor for counter should be set as 0.

I managed to create some code, I'm not sure if it's good solution for this task, for sure there's a way to make it better, or to optimize this, I still learn this, so I just ask you guys for the opinion about it, or maybe some hints.

Sorry for my broken English!

using System;

namespace zaliczenie6test
 {
    class Program
     {
       static void Main(string[] args)
        {
           Counter gate = new Counter();
           char wybor;

            Console.WriteLine("If you want to enter the building(through
            the gate) press 'p', if you want to leave the building press
            'd', if you want check how many people is inside press 'c'
             : ");
        do
        {
            Console.Write("Enter your choice: ");
            wybor = char.Parse(Console.ReadLine());
            if (wybor == 'p')
            {
                Counter gate2 = new Counter();
                gate2.IncNr(wybor);
                gate.Count(wybor);
            }
            else if(wybor == 'd')
            {
                Counter gate3 = new Counter();
                gate3.DecNr(wybor);
                gate.Count(wybor);
            }
        } while (wybor != 'c');
        if (wybor == 'c')
        {
            gate.CheckNr(wybor);
        }

        Console.ReadKey();
    }
}

   class Counter
  {
    private int _licznik;
    private int quantity;
    public int Licznik { get { return _licznik; } set { _licznik =
    value; }
  }

    public Counter()
    {
        _licznik = 0;
    }

    public void IncNr(char wybor)
    {
        if(wybor=='p')
        {
            Licznik += 1;
            Console.WriteLine("The amount of people that just enteredthe
            building through the gate: {0}\n",Licznik);
        }
    }
    public void CheckNr(char wybor)
    {
        if(wybor=='c')
        {
            Console.WriteLine("The current amount of people: {0}",
            quantity);
        }
    }

    public void Count(char wybor)
    {
        if(wybor == 'p')
        {
            quantity += 1;

        }
        else if(wybor == 'd')
        {
            quantity -= 1;
        }
    }

    public void DecNr(char wybor)
    {
        if(wybor == 'd')
        {
            Licznik += 1;
            Console.WriteLine("The amount of people that just passed the
            gate toward the exit is: {0}\n", Licznik);
        }
    }
  }
}

First things first, you need to fix your indentation. It'll make it much easier to follow what's happening. Here's your code properly formatted:

using System;

namespace zaliczenie6test
{
    class Program
    {
        static void Main(string[] args)
        {
            Counter gate = new Counter();
            char wybor;

            Console.WriteLine("If you want to enter the building(through the gate) press 'p', if you want to leave the building press 'd', if you want check how many people is inside press 'c': ");
            do
            {
                Console.Write("Enter your choice: ");
                wybor = char.Parse(Console.ReadLine());
                if (wybor == 'p')
                {
                    Counter gate2 = new Counter();
                    gate2.IncNr(wybor);
                    gate.Count(wybor);
                }
                else if (wybor == 'd')
                {
                    Counter gate3 = new Counter();
                    gate3.DecNr(wybor);
                    gate.Count(wybor);
                }
            } while (wybor != 'c');
            if (wybor == 'c')
            {
                gate.CheckNr(wybor);
            }
            Console.ReadKey();
        }
    }

    class Counter
    {
        private int _licznik;
        private int quantity;
        public int Licznik
        {
            get { return _licznik; }
            set { _licznik = value; }
        }

        public Counter()
        {
            _licznik = 0;
        }

        public void IncNr(char wybor)
        {
            if (wybor == 'p')
            {
                Licznik += 1;
                Console.WriteLine("The amount of people that just entered the building through the gate: { 0}\n", Licznik);
            }
        }
        public void CheckNr(char wybor)
        {
            if (wybor == 'c')
            {
                Console.WriteLine("The current amount of people: {0}", quantity);
            }
        }

        public void Count(char wybor)
        {
            if (wybor == 'p')
            {
                quantity += 1;
            }
            else if (wybor == 'd')
            {
                quantity -= 1;
            }
        }

        public void DecNr(char wybor)
        {
            if (wybor == 'd')
            {
                Licznik += 1;
                Console.WriteLine("The amount of people that just passed the gate toward the exit is: { 0}\n", Licznik);
            }
        }
    }
}

You're also mixing English and Polish - try to stick to one language (Licznik vs quantity).

  The class should contain a methods that will singly increase amount of people or singly decreases amount of them, the third method should show how many is inside already. Last but not least, the constructor for counter should be set as 0.

This is worded a bit ambiguously, but I'll put that down to the translation. From my understanding, it wants you to make three functions

Increment the counter
Decrement the counter
Return the count

It also mentions the difference between the number of entries and the current amount of people in the building, so you might need an additional method to return the total number of entries (ignoring the exits).

You've got those methods, but they're being passed an argument (the choice made). Those methods don't need any argument - they do one thing only. It's the responsibility of the caller to determine which to call (increment, decrement, or show the count). Also, the methods aren't very clear. What is IncNr? What's the Nr represent here? I suspect this may be 'Increment Number', but that's not very clear. Use descriptive method names.

Your Counter class should have the skeleton like so:

class Counter
{
    private int _totalEntrances;
    private int _currentCount;

    public Counter()
    {
        // ...
    }

    public void Increment()
    {
        // ...
    }

    public void Decrement()
    {
        // ...
    }

    public int GetCurrentCount()
    {
        // ...
    }

    public int GetTotalEntries()
    {
        // ...
    }

}

I'll leave it up to you to implement those, but it should be pretty straight forward. Now, for your main method:

if (wybor == 'p')
{
    Counter gate2 = new Counter();
    gate2.IncNr(wybor);
    gate.Count(wybor);
}

Here, you're creating a new counter, increment the count, and then throwing away the counter. This doesn't actually do anything. The only work being done here is gate.Count(wybor). Also be aware that a function named Count() changing a value is very confusing. Judging by the name, it should be returning the count, not changing the count. Also, as we already know it's an entrance, so the code should be:

if (wybor == 'p')
{
    gate.Increment();
}

The same applies to the decrement call.

Currently, your code will exit if the user asks for the count. I'm not sure if this is expected behaviour. I'd expect something like 'q' to quit the program, but 'c' to simply return the count and continue.


Fetching system info using python The following piece of code does ssh do different servers and fetches the system info and display it to the user:

import paramiko

#list variables : ["IP_ADDR", "USERNAME", "PASSWD", "ROOT_PASSWD"]
ip_list = [
    ["192.168.11.44", "root", "****", "****"],
    ["192.168.11.8", "root", "****", "****"],
    ["192.168.11.30", "root", "****", "****"],
    ["192.168.11.6", "****", "****", "****"]
]
os_check_list = ["DISTRIB_DESCRIPTION"]
hard_disks = [
             'sda', 'sdb', 'sdc', 'sdd', 'sde', 'sdf', 'sdg',
             'sdh', 'sdi', 'sdj', 'sdk', 'sdl', 'sdm', 'sdn',
             'sdo', 'sdp', 'sdq', 'sdr', 'sds'
             ]
os = None

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

for ip in ip_list:
    ssh.connect(ip[0], username = ip[1], password = ip[2])

    print("\n \n   ************************************************************ \
           \n   IP ADDR = {}                SYSTEM INFO                           \
          \n  ******************************************************************". format(ip[0]))
    stdin, stdout, stderr = ssh.exec_command("cat /etc/*release")
    stdin.write(ip[3]+"\n")
    for line in stdout.readlines():
       if any(x in line for x in os_check_list):
           os_dist = line.split("=")
           os = os_dist[1]
           print(" Operating System is {}" .format(os))
    if not os:
           stdin, stdout, stderr = ssh.exec_command("cat /etc/system-release")
           for line in stdout.readlines():
               os = line
               print(" Operating System is {}" .format(os))

    os = None

    stdin, stdout, stderr = ssh.exec_command("sudo -k udisksctl status", get_pty = True)
    stdin.write(ip[3]+"\n")
    for line in stdout.readlines():
        if any(x in line for x in hard_disks):
            print(line)
        elif "command not found" in line:
            print("udisksctl not installed on target server")

    stdin, stdout, stderr = ssh.exec_command("sudo dmidecode -t 0 | grep -i version", get_pty = True)
    stdin.write(ip[3]+"\n")
    for line in stdout.readlines():
        if "Version" in line:
            print(line)
        elif "command not found" in line:
            print("dmidecode not installed on target server")

Even though this is working, I felt like I am running too many for loops for fetching the information. How to minimize these many loops?

I would start by making your ip_list easier to understand. You could write a class for this, but using a collectins.namedtuple is a lot easier here:

from collections import namedtuple

Client = namedtuple("Client", "ip username passwd root_passwd")

clients = [Client("192.168.11.44", "root", "****", "****"),
           Client("192.168.11.8", "root", "****", "****"),
           Client("192.168.11.30", "root", "****", "****"),
           Client("192.168.11.6", "****", "****", "****")]

For the hard_disks, you could use the string module for all lowercase letters:

import string

HARD_DISKS = ['sd{}'.format(c) for c in string.ascii_lowercase[:19]]

For your repeated stuff, you could write a function:

def ssh_cmd(cmd, *args, root_passwd=None, **kwargs):
    stdin, stdout, stderr = ssh.exec_command(cmd, *args, **kwargs)
    if root_passwd is not None:
        stdin.write(root_passwd + "\n")
    return stdout.readlines()

This makes the rest slightly nicer to read. However, I would go further and put the other stuff in dedicated functions as well, which can then be put into a utils.py:

import string
from collections import namedtuple

Client = namedtuple("Client", "ip username passwd root_passwd")

HARD_DISKS = ['sd{}'.format(c) for c in string.ascii_lowercase[:19]]

def get_os(client):
    for line in ssh_cmd("cat /etc/*release", root_passwd=client.root_passwd):
       if any(x in line for x in os_check_list):
           # check for the obvious stuff first
           return line.split("=")[1]
    for os in ssh_cmd("cat /etc/system-release"):
       return os

def print_hard_disks(client, hard_disks=HARD_DISKS):
    for line in ssh_cmd("sudo -k udisksctl status", get_pty=True, root_passwd=client.root_passwd):
        if any(hd in line for hd in hard_disks):
            print(line)
        elif "command not found" in line:
            print("udisksctl not installed on target server")

def print_dmi_version(client):
    for line in ssh_cmd("sudo dmidecode -t 0 | grep -i version", get_pty=True, root_passwd=client.root_passwd):
        if "Version" in line:
            print(line)
        elif "command not found" in line:
            print("dmidecode not installed on target server")

It can then be imported in your main script:

import paramiko

from utils import Client, get_os, print_hard_disks, print_dmi_version

BANNER = """

******************************************************************

   IP ADDR = {}                SYSTEM INFO

******************************************************************"""

CLIENTS = [Client("192.168.11.44", "root", "****", "****"),
           Client("192.168.11.8", "root", "****", "****"),
           Client("192.168.11.30", "root", "****", "****"),
           Client("192.168.11.6", "****", "****", "****")]

def main(clients):
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    for client in clients:
        ssh.connect(client.ip, username=client.username, password=client.passwd)

        print(BANNER.format(client.ip))
        print(" Operating System is {}" .format(get_os(client)))
        print_hard_disks(client)
        print_dmi_version(client)

if __name__ == "__main__":
    main(CLIENTS)


How to get the indexes of smallest value per row in a DF, as column index and not column name? I'd like, from a DataFrame, to retrieve a Series which contains the index of the minimal value per row (so axis=1). From the documentation it seems idxmin may be useful
Let' take
import pandas as pd
import numpy as np
df = pd.DataFrame({
    'col1': [0, 10, 2, 30, 4],
    'col2': [2,  3, 4,  5, 6],
    'col3': [9,  7, 5,  3, 1],
})

I've managed to get column names of minimal value
> df.idxmin(axis=1)
0    col1
1    col2
2    col1
3    col3
4    col3
dtype: object

I've managed to get the index of column of minimal vlaue
df.apply(np.argmin, axis=1)
0    0
1    1
2    0
3    2
4    2
dtype: int64

Is there any better way to obtain the 2. result using maybe a full-pandas code ?

df.apply over axis 1 is generally avoid for performance issues. When should I (not) want to use pandas .apply() in my code
1. pd.Index.get_indexer
cols = df.idxmin(axis=1)
idx = df.columns.get_indexer(cols) # return a numpy array
idx_series = pd.Series(idx, index=df.index)
# Here, index=df.index is important when your DataFrame has custom index values

print(idx_series)
0    0
1    1
2    0
3    2
4    2
dtype: int64

2. df.to_numpy and np.ndarray.argmin
idx = df.to_numpy().argmin(axis=1) # returns numpy array
idx_series = pd.Series(idx, index=df.index)
print(idx_series)

0    0
1    1
2    0
3    2
4    2
dtype: int64

Some timeit benchs
from datetime import timedelta
from timeit import timeit

if __name__ == '__main__':

    setup = """
import pandas as pd
import numpy as np
from random import randrange
total = 10000
df = pd.DataFrame({
        'col1': [randrange(10000) for i in range(total)],
        'col2': [randrange(10000) for i in range(total)],
        'col3': [randrange(10000) for i in range(total)],
        'col4': [randrange(10000) for i in range(total)],
        'col5': [randrange(10000) for i in range(total)],
        'col6': [randrange(10000) for i in range(total)],
        'col7': [randrange(10000) for i in range(total)],
    })
    """

    for stmt in ["pd.Series(df.to_numpy().argmin(axis=1), index=df.index)",
                 "pd.Series(df.columns.get_indexer(df.idxmin(axis=1)), index=df.index)",
                 "df.apply(np.argmin, axis=1)"]:
        t = timeit(stmt=stmt, setup=setup, number=20)
        print(f"{str(timedelta(seconds=t)):20s}{stmt}")

# Results
0:00:00.012226      pd.Series(df.to_numpy().argmin(axis=1), index=df.index)
0:00:00.408151      pd.Series(df.columns.get_indexer(df.idxmin(axis=1)), index=df.index)
0:00:23.908125      df.apply(np.argmin, axis=1)


Handling data returned from fetch_assoc() I connected to the db, pull the data using fetch_assoc() (fetch_all(NUMB) is not available on the machine we are working with else this would be less of an issue). I get the returned data and load it into an array. However, it ends up in as an numeric array embedded inside another assoc array. What is the best way to have get the returned data in one, single, simple ASSOC array?

/**
* @Param $conn object [required], $tableName string [required]
* @Return $clmNames array
*/
public function getColumnName($conn = NULL, $tableName = NULL )
{
    if ( !is_object( $conn ) || !is_string( $tableName ) )
    {
        //params are not of correct type. This will eventually throw an error instead of printing to screen
        echo("Error: getColumnName returning false");
        return false;
    }

    //create SQL query
    $stmt = "SELECT column_name FROM information_schema.columns WHERE table_name =  '". $tableName ."' ORDER BY ordinal_position";
    //create query
    $query = $conn->query($stmt);

    //execute query
    $tmp = array();
    while ($row = $query->fetch_assoc())
    {
        //load all returned rows into an array
        $tmp[] = $row;
    }

    //clear memory
    $query->free();unset($query);

    //reCreate Array as a single assoc array instead of a double enbedded one.
    $clmNames = array();
    foreach( $tmp as $tmpKey=>$tmpValue )
    {
        foreach( $tmpValue as $tmpValueKey=>$tmpValueValue )
        {
            $clmNames[] = $tmpValueValue;
        }
    }

    return  $clmNames;
}

The above returns the data correctly. However, I would like to be able to remove the entire foreach dealing with $tmp and $clmNames if possible.

First off, there is no need to check if $conn is an object, simply use the proper procedure for checking if the connection was successful.

if( ! $conn ) {//MySQL
if( $conn->connect_error ) {//MySQLi
try {
    $conn = new PDO( $dsn, $user, $pass );
} catch( PDOException $e ) {
    die( $e->getMessage() );
}

And what is $tableName if not a string? The only other value I see is NULL, in which case you can use an is_null() check instead.

Anyways, now that we are done with that, lets look at the actual problem. The reason you are getting a numerical array of associative arrays is because that is the way you have it coded. You are appending a new array element on to the end of the $tmp array foreach $row in the fetch_assoc() table. Because you used an empty assignment [], this element uses the internal iterator to determine its position in the array, which is always the next available numeric index.

If you want to combine all of these arrays together into one associative array, then you will have to merge them somehow. The easiest way would be to use array_merge(), but then you risk overwriting keys from a previous row. This means if you have identical keys in each row array, then only the last row will be available. This, as you can imagine, would be rather pointless. That leaves us with merging it ourselves. Or does it? Since it appears that you only want the insides of the $row array, then you can just do it all at once, for example:

while( $row = $query->fetch_assoc() ) {
    foreach( $row  AS $value ) {
        $clmNames[] = $value;
    }
}

The above should, if I understand your code correctly, result in the same thing. I'm not sure if its the "best" solution, but it removes two loops, an unnecessary array, and makes your code a little more obvious in what it is trying to do.

A couple more things before I go, try only looping over information once. As shown above, if you find yourself looping over information just to create a new array to loop over it again, then chances are you can do whatever you needed to the first time and avoid having that second loop altogether. Finally, lets look at your foreach loops. If you compare yours to mine, you will notice I am not assigning the key pointer in mine. This is because it is unnecessary unless I am planning on using that value. Avoid assigning variables you are not going to use.

Hope this helps!


Creates a series of folders from a given starting date I made this python script to create dated folders within a directory. This is my first practical automation project and I don't want to cement any bad habits so I was wondering if there is anything I can improve in the code I have written here. Any feedback will be appreciated!
The main challenge I had was to account for months with different day ranges, and then to reset the counts properly for all the variables. I feel like this is more complicated than it probably needs to be but I don't know if there are some python libraries I am unaware of that could help simplify it more.
I thought about having a predefined end date as well but that would require the user to open up a calendar and find 2 dates. I figured by just asking for a start date and how many folders they want, I could make things simpler for the end user but I'm not even sure that that is actually the case.
I also don't know if I've used classes correctly here. At first I had everything thrown into the while statement. I also considered using a for loop instead of while but went with while instead. Again, this is my first real project so any help with other decisions I could have made will be appreciated!
import os
import pathlib

directory = "/Users/myName/Box Sync/SAFAC/Budget Requests/2020-2021/Weekly/"

# directory = input("Enter the path where you would like to create files: ")
date = input("Enter the starting date in mm-dd-yyyy format: ")
period = int(input("Enter the period between each date folder: "))
instances = int(input("How many folders do you want? "))

Folders = []

arguments = date.split("-")
dayArgument = arguments[1]
monthArgument = arguments[0]
yearArgument = arguments[2]
init = instances - instances
elapse = period - period
seperator = "-"

month = int(monthArgument)
day = int(dayArgument) + int(elapse)

def make_directories(input_list):
    for folderName in input_list:
        dirpath = os.path.join(directory, folderName[0:],'Documentation')
        try:
            os.makedirs(dirpath)
        except FileExistsError:
            print('Directory {} already exists'.format(dirpath))
        else:
            print('Directory {} created'.format(dirpath))

def add_folder_to_list(month,day,year,extra):
    full_name_constructor = (month,day,year,extra)
    # full_name_constructor = (str(month),str('0'+str(day)),yearArgument,"Budgets")
    fullName = seperator.join(full_name_constructor)
    Folders.append(fullName)

def format_and_append_folder_name():
    if day < 10 and month >= 10:
        add_folder_to_list(str(month),str('0'+str(day)),yearArgument,"Budgets")
    elif day >= 10 and month < 10:
        add_folder_to_list(str('0'+str(month)),str(day),yearArgument,"Budgets")
    elif day < 10 and month < 10:
        add_folder_to_list(str('0'+str(month)),str('0'+str(day)),yearArgument,"Budgets")
    else:
        add_folder_to_list(str(month),str(day),yearArgument,"Budgets")

def check_month():
    global maxDays
    if (month % 2 ) == 0 and month != 2 : # if even but not February
        maxDays = 30
    elif month == 2 :
        maxDays = 28 #if February
    else :
        maxDays = 31 # if odd

while init < instances :
    check_month()
    format_and_append_folder_name()
    make_directories(Folders)
    # print(Folders[init])            # for debugging

    init += 1
    elapse += period
    day = int(dayArgument) + int(elapse)

    if day > maxDays :
            month += 1
            day -= maxDays                  # reset days
            dayArgument = day               # recalibrate the new zero point for the day to be the date in the new month
            elapse = 0                      # recalibrate elapse point to 0 so it can reiterate the period again

OK let's look at your code.
Unused Imports
pathlib was imported but I don't see any use of it in the codebase, I understand that this might be as a result of testing out approaches to a problem. pylint can help mitigate this by catching unused modules. It also catches some PEP guidelines which your code breaks.
Avoid Globals
You have lots of global variables. A common approach whenever you want to use a global is asking yourself some questions; "How do I express myself with code whilst avoiding this global variable". When in doubt,  prefer objects that holds state to global variables.
Code Review
On first glance,  I can see you are manually parsing dates. This doesn't look good, you introduce bugs when reinventing the wheel. Right now,  your code contains a bug,  it accepts invalid dates,  I can easily create a date like this
date = "12-03-202022"
The above code generates no error, this can be disastrous.
A common alternative is using python  datetime module. This simplify everything, we can move the dates by periods easily,  creating directories becomes quite easy.
NOTE: datetime  module checks for invalid dates.
A simple way of creating a date from a string is given as
from datetime  import datetime
date = datetime.strptime(start_date, "%d/%m/%y")

This simple line of code simplifies everything,  the full code can then be written as
"""Directory creating program"""

import os
from datetime import datetime, timedelta

def next_date(date, period):
    """Increase the date by timedelta
        date: datetime object
        period: timedelta to increase our date by
    """
    new_date = date + timedelta(days=period)
    return '/'.join((str(new_date.day), str(new_date.month), str(new_date.year))), new_date

def make_directories(home_dir, start_date, instances, period):
    """Create sub directories in the home_dir

        home_dir: base directory
        start_date: date to append to directories created
        instances: number of directories to create
        period: difference between two dates
    """
    date = datetime.strptime(start_date, "%d/%m/%y")

    for i in range(instances):
        start_date = start_date.replace('/', '-')
        dirpath = os.path.join(directory, start_date,'Documentation')
        try:
            os.makedirs(dirpath)
        except FileExistsError:
            print('Directory {} already exists'.format(dirpath))
        else:
            print('Directory {} created'.format(dirpath))

        start_date, date = next_date(date, period)

if __name__ == "__main__":
    directory = "/Users/myName/Box Sync/SAFAC/Budget Requests/2020-2021/Weekly/"
    make_directories(directory, '3/1/20', 10, 5)


Robopurging expired files This script uses robocopy purge with list command to out what would be purged it then parses that log file and checks each file and only deletes files older than 7 days. Is there way it could be simplified to maybe use less log files in the process?

$Log = "D:\IT\Backups\Logs\list.txt"
$extrafile = "D:\IT\Backups\Logs\extrafile.txt"
$extradir = "D:\IT\Backups\Logs\extradir.txt"
$extraparsed = "D:\IT\Backups\Logs\extraparsed.txt"
$purged = "D:\IT\Backups\Logs\purged.txt"
$now = Get-Date
$days = "7"
$Lastwrite = $now.AddDays(-$days)
$datetime = get-date -f MM-dd-yy-hhmmtt

If (Test-Path "D:\IT\Backups\Logs\list.txt"){
        Remove-Item "D:\IT\Backups\Logs\list.txt"
}

robocopy "\\10.10.10.9\tcart" "\\192.168.1.44\tcartbkup" /e /Purge /R:1 /W:1 /l /TEE /FP /LOG:"D:\IT\Backups\Logs\list.txt" /MT:16

If (Test-Path $extradir){
        Remove-Item $extradir
}

If (Test-Path $extrafile){
        Remove-Item $extrafile
}

If (Test-Path $extraparsed){
        Remove-Item $extraparsed
}
If (Test-Path $purged){
        rename-item $purged -newname ("purged-" + $datetime + ".txt")
}

Get-Content $Log | Select-String "EXTRA File" | Out-File $extrafile -Width 5000
Get-Content $Log | Select-String "EXTRA Dir" | Out-File $extradir -Width 5000

cat $extradir | %{$_.Substring(28)} | Out-File $extraparsed
cat $extrafile | %{$_.Substring(38)} | Out-File $extraparsed -Append

$files = Get-Content $extraparsed | Select-String "\\" | ForEach-Object {(Get-Item "$_")} | Where {ForEach-Object {($_.LastWriteTime -le "$Lastwrite")}}

foreach ($file in $files)
    {
        if ($File -ne $NULL)
            {
             Write-Host "Deleting File $File" -ForegroundColor "DarkRed"
             Write-Output "Deleting File $File" | Out-File $purged -Append
             Remove-Item $file.FullName -Recurse | Out-Null
             }
             else
             {
             Write-Host "No More files to delete" -ForegroundColor "Green"
             Write-Output "No More files to delete" | Out-File $purged -Append
             }
     }

To reduce the amount of log files I would keep things in memory:

$Log = "C:\scripts\list.txt"
$purged = "C:\scripts\purged.txt"
$now = Get-Date
$days = "7"
$Lastwrite = $now.AddDays(-$days)
$datetime = get-date -f MM-dd-yy-hhmmtt

#Robocopy will automatically overwrite the log file unless you specifiy /log+:
robocopy "c:\downloads" "c:\downloads1" /e /Purge /R:1 /W:1 /l /TEE /FP /LOG:"$log" /MT:16

#Save purged log
rename-item $purged ($purged -replace('.txt', "_$datetime.txt"))

Get-Content $Log | foreach {

    #Split columns by tab
    $fileType =  ( $_ -split("`t") )[1]
    $filePath =  ( $_ -split("`t") )[2]

    if ( ($fileType -like "*Extra File*") -or ($fileType -like "*Extra Dir*") ){
        if ( (Get-Item $filePath).LastWriteTime -le $Lastwrite){
            "Deleting File: $filePath" | Out-File $purged -Append
            remove-item $filePath -WhatIf -Recurse
        }
    }

}


Logging auditing information I had to implement an auditing system where the user have to provide a reason he needs to perform a certain operation. When he provides the reason, he can perform freely it for a certain amount of time. Each execution of the operation needs to be logged somewhere.

I implemented it using a Continuation Passing Style, which allows me to cleanly handle atomically permission checking and operation logging.

public class AuditLogger
{
   private readonly TimeSpan permissionDuration;
   private readonly IAuditDatabase auditDatabase;

   private DateTime permissionExpiry = DateTime.MinValue;
   private string accessReason;

   public AuditLogger(TimeSpan permissionDuration, IAuditDatabase auditDatabase)
   {
       this.permissionDuration = permissionDuration;
       this.auditDatabase = auditDatabase;
   }

   private bool CheckPermission()
   {
       return DateTime.UTCNow < permissionExpiry;
   }

   public void SetAccessReason(string reason)
   {
       reason = reason;
       permissionExpiry = DateTime.UTCNow.Add(permissionDuration);
   }

   public void LogAccessAndExecuteOperation(Action operation, Action onPermissionExpired, Action<Exception> onError)
   {
       if(CheckPermission())
           onPermissionExpired();
       try
       {
           auditDatabase.Log(reason);
           operation();
       }
       catch(Exception exception)
       {
           onError(exception);
       }
   }
}

What do you think about it?

Here it is some client code to show how it is supposed to be used.

public static void Main(string[] args)
{
    var auditLogger = new AuditLogger(TimeSpan.FromMinutes(15), new AuditingDatabase);
    var reason = args[0]; // Let's assume the user tells us what he needs to do at startup
    auditLogger.SetAccessReason(reason);

    // Now the user keeps doing lots of things
    while(1)
    {
        auditLogger.LogAccessAndExecuteOperation(
            () => { /* doing nothing here */ Thread.Sleep(1000); },
            () => {
                Console.WriteLine("Time's up. Can you please tell us again what you need to do?");
                reason = Console.ReadLine();
                auditLogger.SetAccessReason(reason);
            },
            exeption => Console.WriteLine(exception)
            );
    }
}

The class unfortunately violates the Single Responsibility Principle because it has to deal with logging as well as expiring access.

You could use the Decorator Pattern here and split the authorization and logging into separate classes as follows:

public interface IAccessWidget
{
   string AccessReason { get; }
   void SetAccessReason(string reason);
   void ExecuteOperation(Action operation, Action onPermissionExpired, Action<Exception> onError);
}

public class AccessWidget
{
    private readonly TimeSpan _permissionDuration;
    private DateTime _permissionExpiry = DateTime.MinValue;

    public string AccessReason { get; private set; }

    public AccessWidget(TimeSpan permissionDuration)
    {
        _permissionDuration = permissionDuration;
    }

    private bool CheckPermission()
    {
        return DateTime.UtcNow < _permissionExpiry;
    }

    public void SetAccessReason(string reason)
    {
        AccessReason = reason;
        _permissionExpiry = DateTime.UtcNow.Add(_permissionDuration);
    }

    public void ExecuteOperation(Action operation, Action onPermissionExpired, Action<Exception> onError)
    {
        if(CheckPermission())
            onPermissionExpired();
        try
        {
            operation();
        }
        catch(Exception exception)
        {
            onError(exception);
        }
    }
}

public class LoggingAccessWidget : IAccessWidget
{
    private readonly IAuditDatabase _auditDatabase;
    private readonly IAccessWidget _widget;

    public string AccessReason { get { return _widget.AccessReason; } }

    public LoggingAccessWidget(IAccessWidget widget, IAuditDatabase auditDatabase)
    {
        _widget = widget;
        _auditDatabase = auditDatabase;
    }

    public void SetAccessReason(string reason)
    {
       _widget.SetAccessReason(reason);
    }

    public void ExecuteOperation(Action operation, Action onPermissionExpired, Action<Exception> onError)
    {
        _auditDatabase.Log(_widget.AccessReason);
        _widget.ExecuteOperation(operation, onPermissionExpired, onError);
    }
}


Method injection into TestClass local namespace for automatic generation of Python unittest "test_xxx" methods I am writing a series of unit tests in Python 3.5 unittest, which run the exact same test methods on different datasets.  The purpose is to validate proper behavior of each tested function over a range of inputs at different extremes of the likely numerical range of use (large values, small values, badly scaled values, etc.). I would like to construct the TestCase so that it dynamically generates all relevant test_xxx methods for the datasets I've entered.  This way it leaves me less room for typos, and avoids the need to write a raft of new functions every time I add a dataset.

I first wrote the test code with all of my dataset dict objects housed in a simple list, and a single test_the_thing function within my unittest.TestCase subclass.  test_the_thing would iterate over the list of dict datasets, running the test code on each. Two primary problems arose with this approach:

unittest considers the entire execution of test_the_thing to be a single test, and thus I have to search within my datasets to figure out which one failed when the test fails/errors.
When any given test fails/errors in the middle of the iteration, the remainder of the tests are not run.

What I've now got in its place is the following (the code for the entire class can be perused on GitHub):

class TestOpanUtilsVectorProjRejAngle(unittest.TestCase):
    import numpy as np
    from opan.const import OpanEnum

    class DType(OpanEnum):
        V1 = 'V1'
        V2 = 'V2'
        PROJ = 'PROJ'
        REJ = 'REJ'
        ANG = 'ANG'

    class VecType(OpanEnum): # Types of vectors
        O1 = 'O1'   # Both order-one
        LOL = 'LOL' # Both large (large on large)
        SOS = 'SOS' # Both small (small on small)
        LOS = 'LOS' # Large onto small
        SOL = 'SOL' # Small onto large
        BS = 'BS'   # Badly-scaled

    class RelType(OpanEnum): # Type of vector relationship
        NS = 'NS'      # Nonspecific
        PAR = 'PAR'    # Nearly parallel
        NORM = 'NORM'  # Nearly normal
        AP = 'AP'      # Nearly anti-parallel

    namestr = "{0}_{1}"

    # Dict of dicts of data
    data = {
            # Unremarkable vectors with ~order-one components
            namestr.format(RelType.NS, VecType.O1) :
                {DType.V1: np.array([1, 2, 3]),
                 DType.V2: np.array([-1, 3, 8]),
                 DType.PROJ: np.array([-0.391892, 1.175676, 3.135135]),
                 DType.REJ: np.array([1.391892, 0.824324, -0.135135]),
                 DType.ANG: np.float_(25.712002)},
            # ... more data sub-dictionaries are included
            }

    # Template functions
    # Vector projection template
    def template_proj(self, name, data):
        from opan.utils.vector import proj

        v1 = data[self.DType.V1]
        v2 = data[self.DType.V2]
        p = proj(v1, v2)
        for i, t in enumerate(zip(p, data[self.DType.PROJ])):
            self.assertAlmostEqual(*t, delta=1e-6,
                       msg="Test {0}: Index {1}; V1 = {2}; V2 = {3}"
                            .format(name, i, v1, v2))

    # Two more template functions ...

    # Populate the local namespace with the auto-generated
    #  test methods
    for k, d in data.items():
        # Vector projection
        fxnname = "test_Vector_Proj_Good_{0}".format(k)
        fxn = lambda self, k=k, d=d: self.template_proj(k, d)
        locals().update({fxnname: fxn})

    # Populate for the other two template methods by the same approach ...

The test suite runs as expected: if I insert artificial errors in my 'known-good' values in TestOpanUtilsVectorProjRejAngle.data, the respective tests all fail as desired.

Some questions I have include:

Is there a way to achieve this that is better/more secure than locals().update(...)?
Is the lambda the best (or only?) way to encapsulate each function for packaging into the TestOpanUtilsVectorProjRejAngle local namespace?

General recommendations on making the code cleaner, more Pythonic, etc. are also appreciated.

I think your code is pretty good and you are following good practices with the exception to one (common mistake)

for name, datadict in data.items():
     fxnname = "test_data_{0}".format(name)
     fxn = lambda self, n=name, d=datadict: self.template_test_fxn(n, d)
     locals().update({fxnname: fxn})

Should be:

locals_ = locals()
for name, datadict in data.items():
     fxnname = "test_data_{0}".format(name)
     fxn = lambda self, n=name, d=datadict: self.template_test_fxn(n, d)
     locals_.update({fxnname: fxn})

This will prevent locals() from being evaluated when not necessary, but that's not even that big of a deal.

check out for another users use of locals https://stackoverflow.com/questions/8028708/dynamically-set-local-variable-in-python

i encourage other developers to review this, as it is a very interesting project.


HackerRank - Virtual Function Practice This is just simple code to practice the use of virtual functions from a practice challenge in HackerRank. Could you give me some advice about the best practices and style?

This is just a simple exercise from HackerRank

#include <string>
#include <array>
#include <numeric>
#include <iostream>
#include <algorithm>

class Person {
    protected:
    std::string name;
    int age;
    public:
    virtual void getdata() = 0;
    virtual void putdata() const = 0;
};
class Student final : public Person {
    static inline int id_count = 0;
    std::array<int,6> marks;
    int cur_id;
    public:
    Student(){
        cur_id = ++Student::id_count;
    }
    void getdata() override {
        std::cin >> this->name;
        std::cin >> this->age;
        for (auto &mark : marks) {
            std::cin >> mark;
        }
    }
    void putdata() const override{
        std::cout << name << " " << age << " " << std::accumulate(marks.begin(), marks.end(), 0) << " " << cur_id << std::endl;
    }
};

class Professor final : public Person {
    static inline int id_count = 0;
    int publications;
    int cur_id;

    public:
    Professor(){
        cur_id = ++Professor::id_count;
    }
    void getdata() override {
        std::cin >> this->name;
        std::cin >> this->age;
        std::cin >> this->publications;
    }
    void putdata() const override {
        std::cout << name << " " << age << " " << publications << " " << cur_id << endl;
    }
};

Code provided by the question (cannot be edited):

 int main(){
    int n, val;
    std::cin>>n; //The number of objects that is going to be created.
    Person *per[n];

    for(int i = 0;i < n;i++){

        std::cin>>val;
        if(val == 1){
            // If val is 1 current object is of type Professor
            per[i] = new Professor;

        }
        else per[i] = new Student; // Else the current object is of type Student

        per[i]->getdata(); // Get the data from the user.

    }

    for(int i=0;i<n;i++)
        per[i]->putdata(); // Print the required output for each object.

    return 0;

}

Be consistent.
You use a blank line to separate classes Student and Professor: Person and Studentdeserve no less. (Same goes for change in accessibility; alternatives to a blank line include (less) indent.)
Don't write, never present uncommented code.
Use all machine/tool support you can get: have a look at doxygen.
make classes (and functions) final when further derivation/overriding is bound to break something, only.
Think of derivation as extension or specialisation.
You declared common attributes for Person: try and do the same for behaviour, even if it is more cumbersome in C++ (& influenced languages) than in Simula: There, a virtual function gets to call inner - with common code before&after.
In C++, I know no better than starting Derived::f() with Base::f();.
In this example, getdata() & putdata() shouldn't be pure virtual:
Person::getdata() should input name and age; Person::putdata() output them. We are stuck without a method to handle specialised output is complete imposing. One can define void putdata() const { std::cout << toString() << std::endl; } (with toString() facing the exact problem with cur_id "on the other side").
Avoid overly long lines.

name + " " + std::to_string(age)
     + " " + std::accumulate(marks.begin(), marks.end(), 0)
     + " " + std::to_string(cur_id)

is easier to read and probably works just as well as a one-liner.

(While this review has been received exceedingly friendly,
the following code misses the mark correct contemporary C++ by a mile.
(Hopefully closing in with revisions (incorporating advice from Edward's answer);
Having "fun" with IDEone - don't hold your breath.))

Shots from the hip:

/*! \brief Person sporting common attributes
 *         and get/putdata().
 *
 *  Attributes include name, age, and an in-subclass ID.
 *  getdata() reads from std::cin, starting with name&age;
 *  putdata() writes a line to std::cout.
 */
class Person {
 protected:
    //    const int cur_id;
    std::string name;
    int age;
    /*! terse human readable representation.
    */
    virtual std::string toString() const {
        return name + " " + std::to_string(age);
    }
    ~Person(){}
 public:
    /*! read name&age from std::cin.
     */
    virtual void getdata() {
        std::cin >> name >> age;
    }
    /*! putdata() writes a line to std::cout.
     */
    virtual void putdata(std::ostream &out = std::cout) const {
        out << toString() << std::endl;
    }
};

/*! Student adds marks for subjects to the common attributes.
 */
class Student : public Person {
    const int cur_id;
//    static std::atomic<int> id_count(0);
    std::array<int, 6> marks;
 protected:
    std::string toString() const override {
        return Person::toString()
        + " " + std::to_string(
            std::accumulate(marks.begin(), marks.end(), 0))
        + " " + std::to_string(cur_id);
    }
 public:
    static int id_count;// = 0;
    Student() : cur_id(++id_count) {}
    /*! read as Person from std::cin; add marks.
     */
    void getdata() override {
        Person::getdata();
        for (auto &mark : marks)
            std::cin >> mark;
    }
};

/*! Professor adds a publication count to the common attributes.
 */
class Professor final : public Person {
    const int cur_id;
//    static std::atomic<int> id_count(0);
    int publications;
 protected:
    std::string toString() const override {
        return Person::toString()
        + " " + std::to_string(publications)
        + " " + std::to_string(cur_id);
    }
 public:
    static int id_count;// = 0;
    Professor() : cur_id(++id_count) {}
    /*! read as Person from std::cin; add publications.
     */
    void getdata() override {
        Person::getdata();
        std::cin >> publications;
    }
};

int Student::id_count = 0;
int Professor::id_count = 0;


Fibonnaci sequence calculation Below are two ways to calculate an element of the Fibonacci series. Can someone please help me understand why fib1 is so much faster (over 60x) than fib2 ?

fib2 is just an assignment and a decrement whereas fib1 involves a dictionary and recursive calls.

Thanks.

import timeit

def fib1(n, lookup=dict()):
    if n == 0 or n == 1:
        lookup[n] = n

    elif n not in lookup:
        lookup[n] = fib1(n - 1, lookup) + fib1(n - 2, lookup)

    return lookup[n]

def fib2(n):
    n1, n2 = 0, 1
    while n > 1:
        n -= 1
        n1, n2 = n2, n1 + n2
    return n2

n = 1000
recursive = timeit.timeit(lambda: fib1(984), number=n)
iterative = timeit.timeit(lambda: fib2(984), number=n)

print("recursive:", recursive)
print("iterative:", iterative)

"""
output:

recursive: 0.0012962640030309558
iterative: 0.08570116000191774
"""

The key difference is in a technic called "memoization". According to Wikipedia:
In computing, memoization or memoisation is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls and returning the cached result when the same inputs occur again.
So basically, in the first function, you save all the previous results while on the second function you need to calculate everything again from the beginning for every new call.
A quick search on the internet gives this tutorial if you wish to learn more.


Duplicate code in similar classes where I cannot add more inheritance I'm building a .NET Linq-esc query builder for my application and I'm wondering about my implementation and code structure when it comes a few classes.

Basically I'm creating a query builder for SQL, and I need for my classes to be able to using the following behavior:

query.Select(...).From(...).Join(...).Where(...);

I have created 6 Interfaces that contain one method each that donates what can be done to that class E.G the Select class would implement IFromAble which contains the method From(..). This allows the behavior above.

This is the same for all of the other classes above and a few more. Some classes, like Join will implement a few of the interfaces E.g IJoinAble, IWhereAble, IOrderAble, IGroupAble, IHavingAble.

This, as you can imagine will lead to a lot of duplicated code throughout each of my classes.

Each of the classes also implement a common base abstract class, which means I cannot use abstracts to solve the issue.

Is there another way I can implement my intended behavior without all of the duplicated code? Or is this just one of those cases I'm once again tripped up by C#'s single inheritance?

The Join Class

public class ModuleJoin : QueryModule, IJoinAble, IWhereAble, IOrderAble, IGroupAble
{
    public JoinType JoinType { get; set; }
    public QueryField JoinFrom
    {
        get { return (QueryField)this.Fields[0]; }
        set { this.Fields[0] = value; }
    }
    public QueryField JoinTo
    {
        get { return (QueryField)this.Fields[1]; }
        set { this.Fields[1] = value; }
    }

    public ModuleJoin(Query query, JoinType joinType, QueryField joinFrom, QueryField joinTo)
    {
        this.PerentQuery = query;
        this.Fields.Clear();
        this.Fields.Add(joinFrom);
        this.Fields.Add(joinTo);
        JoinType = joinType;
    }

    public ModuleJoin Join(JoinType joinType, QueryField joinFrom, QueryField joinTo)
    {
        ModuleJoin join = new ModuleJoin(this.PerentQuery, joinType, joinFrom, joinTo);
        this.PerentQuery.ModuleJoins.Add(join);
        return join;
    }

    public ModuleWhere Where(IFieldPart itemOne, Comparator comparator, IFieldPart itemTwo)
    {
        ModuleWhere where = new ModuleWhere(this.PerentQuery, itemOne, comparator, itemTwo);
        this.PerentQuery.ModuleWheres.Add(where);
        return where;
    }

    public ModuleGroupBy GroupBy(params IFieldPart[] fields)
    {
        ModuleGroupBy groupBy = new ModuleGroupBy(this.PerentQuery, fields);
        this.PerentQuery.ModuleGroupBy = groupBy;
        return groupBy;
    }
}

QueryModule (The interface on this just has those properties)

public abstract class QueryModule : IQueryModule
{
    public Query PerentQuery { get; set; }
    public FieldCount FieldCount { get; set; }
    public List<IFieldPart> Fields { get; set; }
    public IFieldPart Field
    {
        get { return Fields[0]; }
        set
        {
            if (Fields.Count == 1)
                Fields[0] = value;
            else
                Fields.Add(value);
        }
    }

    public QueryModule()
    {
        this.Fields = new List<IFieldPart>();
    }
}

IJoinAble (An example of one of the interfaces)

interface IJoinAble
{
    ModuleJoin Join(JoinType joinType, QueryField joinFrom, QueryField joinTo);
}

Interesting question. The most simple and direct way to approach this problem (not necessarily the wisest) - implement Select(), From(), Join() etc. as extension methods on the interfaces ISelectable, IFromable, IJoinable, etc. The ability to write extension methods to interfaces is a powerful and fascinating language feature with many implications...


Automate tasks with Lambdas Can I use #define fori(i,n) for(int i=0;i<=n;i++) in Java?

And of course, Java doesn't have precompiler so we cannot do that in java. So I thought we can do that fori method either using stream() or manually defining a method. But I thought instead of just traversing why can't I put the third parameter as lambda expression that operates.

Functional interface:

interface func
{
    int calculate(int a,int b);
}

Method implementation:

 int run(int[] array,int start,int end,func t)
 {
   int result =0;
   for(int i=start;i<=end;i++)
   {
     System.out.println(result);
     result = t.calculate(array[i], result);
    }
    return result;
  }

Usage:

public static void main(String[] args)
{
        int[] array = {1,2,3,4,5,3,5,3,2,2,3,2,23,2};
        System.out.println(run(array, 0, 5, (a,b) -> a + b  ));
}

I would like a review on this code pattern. My major concern is that how much complicated lambda expression I can use here. Also, what is the future scope of this what could be done with this pattern?

The pattern you have implemented is basically a reduction or foldLeft method.
Compare it to the Stream#reduce method in Java, it is very similar.

To answer your question about the complexity of the lambda to put into the method: There is no limit. You can do everything you like in there. A lambda expression is nothing more than an implementation of the single-method interface you have created. And there is no limit on the complexity of a class implementing an interface.

Now for some real code review:

Use proper and understandable names and use built-in types

Your interface is simply named func. This probably means "function" but you don't have to save on characters, Java is already quite verbose.
What you have, is a special kind of BiFunction or more precisely, BinaryOperator. Now I know that generics don't allow you to specify primitive types for the generic arguments, but your func interface is not necessary and  can be replaced by a BinaryOperator<Integer>.

Your run method has a very generic name. Sure, there is code that is run, but it doesn't explain, what this method does.
It would be more precise to call your method reduce or foldLeft or something similar, that explains in the name, what this method does.

Make your method do one thing

Currently your method calculates a result and, in addition to that, prints the value of result for every iteration. What if you or someone who uses your method does not want to have every single intermediate result printed to them?

If you want to print the intermediate values, do so in your lambda.

You wanted to think about the future

Your method currently only works with primitive integer arrays. Nothing stops you from making it work with generic types or even lists.
And if you go that far, you'll probably use streams anyway.


Efficiently concatenate substrings of long list of strings I am having performance problems with the following python function:

def remove_comments(buffer):
    new_buffer = ''
    lines = buffer.split('\n')
    for line in lines:
        line_wo_comments = line.split('--')[0] + '\n'
        new_buffer = new_buffer + line_wo_comments
    return new_buffer

When buffer is very large (thousands+ lines), the function gets slower and slower as it processes the buffer.

What techniques could I use to speed this function call up?

Assume that the input is a source code file. Lines of length 1 - ~120 characters. Lines may or may not have comments. The files could be many lines long. The especially problematic ones are machine generated (1-10k+ lines long).

Update: The intention is to use this as a "pre-processing" step for the buffer contents (a file). I guess I am not too interested in possibly ways to completely refactor this (i.e. methods to avoid needing to iterate through all the lines multiple times), but rather make the essence of buffer in / buffer out as fast as possible.

The Performance Tips section at python.org has comments about doing repeated string concatenation which you may find here:

https://wiki.python.org/moin/PythonSpeed/PerformanceTips#String_Concatenation

Specifically, it suggests using "".join(listOfStrings) instead of repeatedly appending to an accumulator with +=.

So I would try something like this, using re.finditer() to find all of the comments, and place the non-comment parts into a list:

import re

def removeComments(s):
  chunks = []
  offset = 0
  for m in re.finditer("--.*\n", s):
    chunks.append( s[offset: m.start(0)] )
    offset = m.end(0)-1
  chunks.append( s[offset:] )
  return "".join(chunks)

s = """
line 1
line 2  -- comment 2
line 3
line 4 -- comment 4
line 5
line 6 -- comment 6
line 7
"""
print removeComments(s)

An advantage of this approach over splitting each line is that if there are large chunks of your program which do not have any comments they will transferred to the chunks list in one piece instead of as separate lines.

Update

I would also try using a regexp replace approach - it could be even faster:

def removeComments(s):
  return re.sub('(?m)--.*$', '', s)


Rotate List by K places I am trying to solve the question at

:https://leetcode.com/problems/rotate-list/

Question:

  Given a linked list, rotate the list to the right by k places, where k
  is non-negative.

  Example 1:

  Input:  1->2->3->4->5->NULL, k = 2
  Output: 4->5->1->2->3->NULL

  Explanation:

  rotate 1 steps to the right: 5->1->2->3->4->NULL
  rotate 2 steps to the right: 4->5->1->2->3->NULL

//public class ListNode
//{
//    public int val;
//    public ListNode next;
//    public ListNode(int x) { val = x; }
//}

//k = number of places
//head =Given Linked List

    public  ListNode RotateRight(ListNode head, int k) {
        if (k > 0 && head != null) {
            ListNode tail = head,
            RotatedList = null,
            kthnode = head,
            kthPrevNode = head;
            int listLength = 0;
            while (tail != null) {
                listLength += 1;
                tail = tail.next;
            }

            k = k % listLength;
            if (k == 0 || listLength == 0) {
                return head;
            }

            for (int i = 0; i < listLength - k; i++) {

                kthPrevNode = kthnode;

                kthnode = kthnode.next;

            }

            RotatedList = kthnode;
            kthPrevNode.next = null;
            while (kthnode.next != null) {
                kthnode = kthnode.next;
            }

            kthnode.next = head;
            return RotatedList;

        }

        return head;
    }
}
}

I have passed all test cases but I am looking to improve code efficiency and time complexity,Kindly let me know if there are any bad practices as well

if (k > 0 && head != null) {
            ...
            return RotatedList;

        }

        return head;

would be clearer (and more consistent with the other special case) as

        if (k <= 0 || head == null) {
            return head;
        }

        ...
        return RotatedList;

(although really if k < 0 I think it should throw new ArgumentOutOfRangeException(nameof(k))).

            ListNode tail = head,
            RotatedList = null,
            kthnode = head,
            kthPrevNode = head;

Most of these don't need to be declared so early. Declaring variables as late as possible (and, more generally, in the narrowest scope possible) helps to reduce cognitive load when maintaining the code.

            int listLength = 0;
            while (tail != null) {
                listLength += 1;
                tail = tail.next;
            }

This would be worth factoring out as a separate method Length(ListNode head).

            k = k % listLength;
            if (k == 0 || listLength == 0) {
                return head;
            }

This is sort-of buggy. If listLength == 0 then k % listLength will throw an ArithmeticException. However, you can never reach here in that case, because listLength == 0 is equivalent to head == null, and that was already handled in the first special case. To remove confusion, delete || listLength == 0. If you want to validate that condition, insert System.Diagnostics.Debug.Assert(listLength > 0); before the % line.

            for (int i = 0; i < listLength - k; i++) {

                kthPrevNode = kthnode;

                kthnode = kthnode.next;

            }

It's not actually necessary to track two nodes here. The invariant is that kthPrevNode.next == kthnode, so it suffices to track kthPrevNode.

Also, the blank lines seem excessive to me.

            RotatedList = kthnode;

It's conventional in C# that local variable names start with a lower case letter, so to avoid confusion I would rename RotatedList to rotatedList (or perhaps rotatedHead).

There are only two things where I can see that the efficiency can be improved: one is eliminating kthnode in the loop, as commented above; the other would be to hang on to the last node when finding the length, so that you don't need to find it a second time in the last few lines. (That would imply changing the factored out method to return an (int, ListNode) tuple).

As for time complexity, it's already optimal: linear time. Good job!


Number of Islands in a 2d grid Recently, I've solved this "Number of Islands" problem on LeetCode, the solution was accepted by the LeetCode OJ.

Problem Description

  Given a 2d grid map of '1's (land) and '0's (water), count the number
  of islands. An island is surrounded by water and is formed by
  connecting adjacent lands horizontally or vertically. You may assume
  all four edges of the grid are all surrounded by water.

Examples/Test Cases

A single island:

11110
11010
11000
00000

3 islands:

11000
11000
00100
00011

The Solution

The idea behind the solution posted below is to:

iterate over every cell of the grid
when find a 1 value, increment the island counter, use the BFS to find all cells in the current island
mark all the cells in the current island with value 2

The Code:

from collections import deque

class Solution(object):
    def append_if(self, queue, x, y):
        """Append to the queue only if in bounds of the grid and the cell value is 1."""
        if 0 <= x < len(self.grid) and 0 <= y < len(self.grid[0]):
            if self.grid[x][y] == '1':
                queue.append((x, y))

    def mark_neighbors(self, row, col):
        """Mark all the cells in the current island with value = 2. Breadth-first search."""
        queue = deque()

        queue.append((row, col))
        while queue:
            x, y = queue.pop()
            self.grid[x][y] = '2'

            self.append_if(queue, x - 1, y)
            self.append_if(queue, x, y - 1)
            self.append_if(queue, x + 1, y)
            self.append_if(queue, x, y + 1)

    def numIslands(self, grid):
        """
        :type grid: List[List[str]]
        :rtype: int
        """

        if not grid or len(grid) == 0 or len(grid[0]) == 0:
            return 0

        self.grid = grid

        row_length = len(grid)
        col_length = len(grid[0])

        island_counter = 0
        for row in range(row_length):
            for col in range(col_length):
                if self.grid[row][col] == '1':
                    # found an island
                    island_counter += 1

                    self.mark_neighbors(row, col)

        return island_counter

if __name__ == '__main__':
    grid = """11000
    11000
    00100
    00011"""

    grid = [list(line) for line in grid.splitlines()]
    print(Solution().numIslands(grid))

The Questions:

Is it the most optimal solution to the problem, or is there a more efficient approach? What would you improve code-quality or code-organization wise?

One way to simplify the problem is a shift of datastructure to hold your islands. By converting the grid to a set containing only the "land" parts, you can speed up the computation by entirely removing the check for "does it fit into the map". The idea here is to pick a land in your set — doing so you'll know that you have discovered a new island — and remove all land connected to it in your set. Luckily set.remove is easy to use and let us know if the element was present or not so we can stop the recursion.

I would also advise against using explict len to iterate over the grid. enumerate should give you anything you need in a more cleaner way.

Lastly, in the testing code, grid is

[['1', '1', '0', '0', '0'],
 [' ', ' ', ' ', ' ', '1', '1', '0', '0', '0'],
 [' ', ' ', ' ', ' ', '0', '0', '1', '0', '0'],
 [' ', ' ', ' ', ' ', '0', '0', '0', '1', '1']]

So you might have some inexpected results. I suggest the use of the textwrap module like so:

if __name__ == '__main__':
    from textwrap import dedent
    grid = dedent("""\
        11000
        11000
        00100
        000111""")
    ...

Revised algorithm would look like:

class Solution(object):
    def numIslands(self, grid):
        """
        :type grid: List[List[str]]
        :rtype: int
        """

        # Convert the grid to a bag of land
        self.lands = {
            (row, col)
            for row, line in enumerate(grid)
            for col, land in enumerate(line)
            if land == '1'
        }

        island_counter = 0
        # Remove islands from the bag one by one
        while self.lands:
            island_seed = next(iter(self.lands))  # pick a random land
            island_counter += 1
            self.remove_island(island_seed)

        return island_counter

    def remove_island(self, coordinates):
        try:
            self.lands.remove(coordinates)
        except KeyError:
            return

        x, y = coordinates
        self.remove_island((x - 1, y))
        self.remove_island((x + 1, y))
        self.remove_island((x, y - 1))
        self.remove_island((x, y + 1))

If you don't like remove_island being recursive, you can still write it as a BFS as you did in your solution. But since you’re not poping from the left, there is no need of a deque:

    def remove_island(self, coordinates):
        queue = [coordinates]
        while queue:
            x, y = queue.pop()
            try:
                self.lands.remove((x, y))
            except KeyError:
                pass
            else:
                queue += [(x - 1, y), (x + 1, y), (x, y - 1), (x, y + 1)]


Make this implementation of counting sort Pythonic This is my crude attempt at implementing a counting sort algorithm. My goal is to be able to sort lists of ints, negative and positive. As you can see I've employed such hacks as return res[1:] to get by. I'd appreciate any feedback I can get to make this a more pythonic implementation.

def find_min(a_list):
    min = a_list[0]
    for item in a_list:
        if item < min:
            min = item
    return min

def find_max(a_list):
    max = a_list[0]
    for item in a_list:
        if item > max:
            max = item
    return max

def counting_sort(a_list):
    min = find_min(a_list)
    max = find_max(a_list)

    counts = [0] * (abs(max) + abs(min) + 1)

    for i in range(0, len(a_list), 1):
        counts[a_list[i] - min] += 1

    sum = 0
    for j in range(0, len(counts), 1):
        sum += counts[j]
        counts[j] = sum

    res = [0] * (len(a_list)+1)
    for k in range(0, len(a_list), 1):
        res_index = counts[a_list[k]-min]
        res[res_index] = a_list[k]
        counts[a_list[k]-min] -= 1

    return res[1:]

My two cents:

You can get rid of def find_min(a_list): and def find_max(a_list):. Instead, use python's built-ins min() and max():

  min(iterable[, key])

  Return the smallest item in an iterable
  or the smallest of two or more arguments.

  If one positional argument is provided, iterable must be a non-empty
  iterable (such as a non-empty string, tuple or list). The smallest
  item in the iterable is returned. If two or more positional arguments
  are provided, the smallest of the positional arguments is returned.

  max(iterable[, key])

  Return the largest item in an iterable or the
  largest of two or more arguments.

  If one positional argument is provided, iterable must be a non-empty
  iterable (such as a non-empty string, tuple or list). The largest item
  in the iterable is returned. If two or more positional arguments are
  provided, the largest of the positional arguments is returned.

So you'll have:

...
min_element, max_element = min(a_list), max(a_list)
...

Now, regarding you algorithm, you can avoid those hacks you were talking about by rewriting your function as:

def count_sort(array):
    min_element, max_element = min(array), max(array)
    count_array = [0] * (max_element - min_element + 1)

    for val in array:
        count_array[val - min_element] += 1

    sorted_array = []
    for i in range(min_element, max_element + 1):
        if count_array[i - min_element] > 0:
            for j in range(0, count_array[i - min_element]):
                sorted_array.append(i)

    return sorted_array

More, as @ChatterOne mentioned, instead of the final for loop with append, you could do:

sorted_array.extend([i] * count_array[i - min_element])

So, your final code would look like this:

def count_sort(array):
    min_element, max_element = min(array), max(array)
    count_array = [0] * (max_element - min_element + 1)

    for val in array:
        count_array[val - min_element] += 1

    sorted_array = []
    for i in range(min_element, max_element + 1):
        if count_array[i - min_element] > 0:
            for j in range(0, count_array[i - min_element]):
                sorted_array.append(i)

    return sorted_array

if __name__ == '__main__':
    my_array = [3, 2, -1, 1, 5, 0, 10, 18, 25, 25]
    print(count_sort(my_array))

or

def count_sort(array):
    min_element, max_element = min(array), max(array)
    count_array = [0] * (max_element - min_element + 1)

    for val in array:
        count_array[val - min_element] += 1

    sorted_array = []
    for i in range(min_element, max_element + 1):
        if count_array[i - min_element] > 0:
            sorted_array.extend([i] * count_array[i - min_element])

    return sorted_array

if __name__ == '__main__':
    my_array = [3, 2, -1, 1, 5, 0, 10, 18, 25, 25]
    print(count_sort(my_array))

Extras:

You can see that I also added if __name__ == '__main__'. By doing the main check, you can have that code execute only when you want to run the module as a program, and not have it execute when someone just wants to import your module and call your functions themselves.

I also changed the names min and max(which are already built-ins in Python) to min_element and max_element, respectively.

And the result:

[-1, 0, 1, 2, 3, 5, 10, 18, 25, 25]


Recursively listing the content of an FTP server Today I found myself in need of programmatically getting a recursive list of the files on a remote FTP server to look for some data archives. Much to my surprise, I could not find this functionality implemented in some kind of Python library. So I decided to write a simple recursive version of ftplib.mlsd, that can be found below:

ftphelper.py

def recursive_mlsd(ftp_object, path="", maxdepth=None):
    """Run the FTP's MLSD command recursively

    The MLSD is returned as a list of tuples with (name, properties) for each
    object found on the FTP server. This function adds the non-standard
    property "children" which is then again an MLSD listing, possibly with more
    "children".

    Parameters
    ----------
    ftp_object: ftplib.FTP or ftplib.FTP_TLS
        the (authenticated) FTP client object used to make the calls to the
        server
    path: str
        path to start the recursive listing from
    maxdepth: {None, int}, optional
        maximum recursion depth, has to be >= 0 or None (i.e. no limit).

    Returns
    -------
    list
        the recursive directory listing

    See also
    --------
    ftplib.FTP.mlsd : the non-recursive version of this function
    """
    if maxdepth is not None:
        maxdepth = int(maxdepth)
        if maxdepth < 0:
            raise ValueError("maxdepth is supposed to be >= 0")

    def _inner(path_, depth_):
        if maxdepth is not None and depth_ > maxdepth:
            return

        inner_mlsd = list(ftp_object.mlsd(path=path_))
        for name, properties in inner_mlsd:
            if properties["type"] == "dir":
                rec_path = path_+"/"+name if path_ else name
                res = _inner(rec_path, depth_+1)
                if res is not None:
                    properties["children"] = res

        return inner_mlsd

    return _inner(path, 0)

A very basic example usage could be like:

import ftplib
import getpass
import pprint
import sys

import ftphelper

host = "ftp1.at.proftpd.org"   # from the Python docs
directory = "historic"
print(f"Login to '{host}' to list '{directory}'", file=sys.stderr)

default_user = ""   # getpass.getuser() could also be used where appropriate
prompt = f"Username (default: '{default_user}'): "
if sys.stdout.isatty():
    user = input(prompt)
else:
    print(prompt, end="", file=sys.stderr)
    user = input()
user = user.strip()
user = user if user else default_user

with ftplib.FTP(host=host) as ftp:
    ftp.login(user=user, passwd=getpass.getpass())
    listing = ftphelper.recursive_mlsd(ftp, directory)

pp = pprint.PrettyPrinter(indent=2)
pp.pprint(listing)

As always, every kind of feedback is welcome and much appreciated.

Seems good except for a few small improvements:

path_+"/"+name is better perceived with f-string formatting: f'{path_}/{name}
user if user else default_user is a verbose version of user or default_user


Dynamic array container This is primarily a container for quicksort and mergesort:

#include "c_arclib.cpp"
template <class T> class dynamic_array
  {
  private:
    T* array;
    T* scratch;
  public:
    int size;
    dynamic_array(int sizein)
      {
      size=sizein;
      array = new T[size]();
      }
    void print_array()
      {
      for (int i = 0; i < size; i++) cout << array[i] << endl;
      }
    void merge_recurse(int left, int right)
      {
      if(right == left + 1)
        {
        return;
        }
      else
        {
        int i = 0;
        int length = right - left;
        int midpoint_distance = length/2;
        int l = left, r = left + midpoint_distance;
        merge_recurse(left, left + midpoint_distance);
        merge_recurse(left + midpoint_distance, right);
        for(i = 0; i < length; i++)
          {
          if((l < (left + midpoint_distance)) && (r == right || array[l] > array[r]))
            {
            scratch[i] = array[l];
            l++;
            }
          else
            {
            scratch[i] = array[r];
            r++;
            }
          }
        for(i = left; i < right; i++)
          {
          array[i] = scratch[i - left];
          }
        }
      }
    int merge_sort()
      {
      scratch = new T[size]();
      if(scratch != NULL)
        {
        merge_recurse(0, size);
        return 1;
        }
      else
        {
        return 0;
        }
      }
    void quick_recurse(int left, int right)
      {
      int l = left, r = right, tmp;
      int pivot = array[(left + right) / 2];
      while (l <= r)
        {
        while (array[l] < pivot)l++;
        while (array[r] > pivot)r--;
        if (l <= r)
          {
          tmp = array[l];
          array[l] = array[r];
          array[r] = tmp;
          l++;
          r--;
          }
        }
      if (left < r)quick_recurse(left, r);
      if (l < right)quick_recurse(l, right);
      }
    void quick_sort()
      {
      quick_recurse(0,size);
      }
    void rand_to_array()
      {
      srand(time(NULL));
      int* k;
      for (k = array; k != array + size; ++k)
        {
        *k=rand();
        }
      }
  };
int main()
  {
  dynamic_array<int> d1(10);
  cout << d1.size;
  d1.print_array();
  d1.rand_to_array();
  d1.print_array();
  d1.merge_sort();
  d1.print_array();
  }

My first comment is its named badly.

dynamic_array implies that I can use [] operator on it and get a value out.

You have owned RAW pointers in your structure.

private:
  T* array;
  T* scratch;

First this means you need to look up RAII to make sure these members are correctly deleted.

Second you you need to look up the rule of three (or 5 in C++11) to make sure they are copied correctly.

You have owned RAW pointers in your structure. This means you need to correctly manage the object as a resource. This means constructions/destruction/copy (creation and assignment) need to be taken care of correctly.

Either do this manually or use a standard container that will do it for you. I suggest a standard container.

void print_array()
  {
  for (int i = 0; i < size; i++) cout << array[i] << endl;
  }

If you are going to write print_array at least write it so that it can use alternative stream (not just std::cout). Then write the output operator.

std::ostream& operator<<(std::ostream& stream, dynamic_array const& data)
{
     data.print_array(stream); // After you fix print_array
     return stream;
}

Also note that a method that access data but does not modify the state of the object should be marked const. So the signature should be: void print_array() const

Are the following members really part of the array?

void merge_recurse(int left, int right)
int merge_sort()
void quick_recurse(int left, int right)

OK. Lets assume they are for now.
Then  void merge_recurse(int left, int right) should be a private member. There should be no reason to call this from externally.

  scratch = new T[size]();
  if(scratch != NULL)

scratch will Never be NULL.

I think merge (in merge_recurse) is easier to write than you are making it:

    int index = 0;
    int l = left;
    int r = midpoint;
    while((l < midpoint) && (r < right))
    {
      scratch[index++] = (array[l] > array[r]))
                           ? array[l++]
                           : array[r++];
    }
    // One of the two ranges is empty.
    // copy the other into the destination.
    while(l < midpoint)
    {
        scratch[index++] = array[l++];
    }
    while(r < right)
    {
        scratch[index++] = array[r++];
    }

You should only call srand() once in an application:

void rand_to_array()
  {
  srand(time(NULL));

By putting srand() inside the structure you are opening it up to be called multiple times. Call it once just after main() then don't call it again.

When you can use the standard tools:

      tmp = array[l];
      array[l] = array[r];
      array[r] = tmp;

Can be replaced with:

 std::swap(array[l], array[r]);

I am relatively sure these two are wrong:

  while (l <= r)
    if (l <= r)

They should be:

  while (l < r)
    if (l < r)


Median of two sorted arrays (optimal runtime) I've written a solution to the leetcode problem Median of Two Sorted arrays. The solution I anticipated coming up with initially was going to features some kind of binary search on both arrays based on the range of numbers (given that the question calls for a runtime of O(log (m+n))).

Along the way, I thought of another, simpler solution that featured averaging the largest number of both arrays (the last element of one of the arrays) and the smallest number of the two arrays (index 0 of one of the arrays). I also include a number of if statements to handle the possible different array lengths that could be passed in. I'm particularly interested in the approach of this solution since it would have a faster runtime than what was suggested.

This solution surprisingly passes 2056 / 2084 of the test cases. It fails, however, on the input arrays of [3],[-2,-1], which leads me to believe that the code will fail only when there exist negative numbers in either array.  Is this the case, or is it just that the test cases used for the question aren't wide-ranging enough?

class Solution:
    def findMedianSortedArrays(self, nums1, nums2):
        """
        :type nums1: List[int]
        :type nums2: List[int]
        :rtype: float
        """
        if len(nums1) == 0 and len(nums2) == 0:
            return None
        if len(nums1) == 0 and len(nums2) > 0:
            if len(nums2) == 1:
                return nums2[0]
            elif len(nums2) > 1 and len(nums2) % 2 == 1:
                lo,hi = 0,len(nums2)-1
                mid = (hi+lo)//2
                return nums2[mid]
            elif len(nums2) > 1 and len(nums2) % 2 == 0:
                lo,hi = 0,len(nums2)-1
                mid = (hi+lo)//2
                return (nums2[mid] + nums2[mid+1])/2

        if len(nums1) > 0 and len(nums2) == 0:
            if len(nums1) == 1:
                return nums1[0]
            elif len(nums1) > 1 and len(nums1) % 2 == 1:
                lo,hi = 0,len(nums1)-1
                mid = (hi+lo)//2
                return nums1[mid]
            elif len(nums1) > 1 and len(nums1) % 2 == 0:
                lo,hi = 0,len(nums1)-1
                mid = (hi+lo)//2
                return (nums1[mid] + nums1[mid+1])/2

        l1A,l2A,l1B,l2B = 0,0,len(nums1)-1,len(nums2)-1
        arrBeg = nums1[l1A] if nums1[l1A] <nums2[l2A] else nums2[l2A]
        arrEnd = nums1[l1B] if nums1[l1B] > nums2[l2B] else nums2[l2B]
        target = (arrEnd+arrBeg)/2
        return target

I believe the test cases are not generated randomly. Your code will fail on almost all cases if the cases are generated in a random way.
For instance,
if array A is [0, 10000], and array B contains an arbitrary number of elements with arbitrary values from (0, 10000) except 5000, your code will fail. It is not relevant if the test cases contain negative numbers or not.