High Speed Python Web Scraper Optimised I needed a lot of data for a tensorflow project so I made a web scraper to get all of the text and links off of websites then to repeat the process at all of those links.

I left it on overnight and it did not get much done, so I spent the day optimizing it. I can't find any ways to optimize it more (If anyone knows a way I would like to hear it.) I used it on CNN and now I have a 9 gig text file.

BTW: I used faster_than_requests and selectolax because they are faster then urllib3 and bs4 and you should check them out

import cython # helps speed up code
from selectolax.parser import HTMLParser #bs4 but faster
import faster_than_requests #urllib but faster
import _pickle as pickle #saving code
from colorama import init #just makes error messages stand out
from colorama import Fore, Back, Style

init() #colorama thing

#cdef is a cython thing, helping speed up code
cdef int i = 0

cdef list urls
cdef list text

cdef set visits #is a set for efficiency

cdef str mainsite = "https://www.cnn.com" #The main site keeps the scraper
                     #from straying too far from its original site.
cdef str source

parsing = True

try:
  with open('visits.pickle', 'rb') as f:
    visload = pickle.load(f)
    visits = visload[1]
    i = visload[0]
except Exception as e:
  print(Back.RED + "Error loading Visits: " + str(e))
  visits = set('')
  i = 0

try:
  with open('txt.pickle', 'rb') as f:
    txt = pickle.load(f)
except Exception as e:
  print(Back.RED + "Error loading txt: " + str(e))
  txt = []

try:
  with open('links.pickle', 'rb') as f:
    urls = pickle.load(f)
except Exception as e:
  print(Back.RED + "Error loading urls: " + str(e))
  urls = ["https://www.cnn.com"]

while parsing:

  try:
    if urls[0][0] == "/": #checks to see if it can go
                          # to the site directly or it needs to add
                          #the main site to the front
      source = faster_than_requests.get2str(mainsite + urls[0])
      dom = HTMLParser(source)
      print(Back.BLACK + mainsite + urls[0])
    else:
      source = faster_than_requests.get2str(urls[0])
      dom = HTMLParser(source)
      print(Back.BLACK + urls[0])

    for tag in dom.tags('p'):
      txt.append(str(dom.text())) #finds text and saves it

    for tag in dom.tags('a'):
      attrs = tag.attributes
      if 'href' in attrs:
          urls.append(attrs['href']) #finds links and saves them
  except:
    print(Back.RED + f"Error: {urls[0]}") # it will through up an error
                                          # if it tries to go to a sub-page
                                          # of another site, but this is
                                          # an intended feature

  visits.add(urls[0])
  #visits keeps track of visites web pages

  i = i + 1
  clean = True

  #clean make shure that it does not repeat a webpage.
  while clean:
    if urls[0] in visits:
      del(urls[0])
    else:
      clean = False

  print(Back.BLACK + f"urls:{len(urls)}, i:{i}, text lang:{len(txt)}")

  if i % 10000 == 0:
    #Save every 10000 webpages
    with open('txt.pickle', 'wb') as f:
      pickle.dump(txt, f, pickle.HIGHEST_PROTOCOL)
    with open('links.pickle', 'wb') as f:
      pickle.dump(urls, f, pickle.HIGHEST_PROTOCOL)
    with open('visits.pickle', 'wb') as f:
      pickle.dump([i, visits], f, pickle.HIGHEST_PROTOCOL)

  if 0 == len(urls):
    parsing = False
    print(txt)
    with open('txt.pickle', 'wb') as f:
      pickle.dump(txt, f, pickle.HIGHEST_PROTOCOL)
    with open('links.pickle', 'wb') as f:
      pickle.dump(urls, f, pickle.HIGHEST_PROTOCOL)
    with open('visits.pickle', 'wb') as f:
      pickle.dump([i, visits], f, pickle.HIGHEST_PROTOCOL)

If you want absolute performance

You could avoid printing anything at all. Since that can be slow in some cases. If you absolutly need to know at least what happened you could try flushing at the end of the process.
I know with python is delicated, but you could try visiting multiple pages at the same time? Threads or the equivalent in python


Conditional statements relating to hospital management It works just fine, but is there any other way to write this shorter?

Nurses are only available to intensive care patients (room I) and TV's and telephones are only available to non intensive care patients (room D or room P). Also, X_COST has a value of 0.

if (room.equals("I")) { // if (room.equals("I"))

    roomCost = 395 * (double)days;
    //System.out.println(roomCost);
    TVCost = X_COST;
    //System.out.println(TVCost);
    phoneCost = X_COST;
    //System.out.println(phoneCost);

    if (nurse.equals("X"))

        nurseCost = X_COST;
        //System.out.println(nurseCost);

    else if (room.equals("N"))

        nurseCost = 250 * (double)days;
        //System.out.println(nurseCost);

    else

        nurseCost = 275 * (double)days;
        //System.out.println(nurseCost);

} // if (room.equals("I"))

else if (room.equals("P")) { // else if (room.equals("P"))

    roomCost = 350 * (double)days;
    //System.out.println(roomCost);
    nurseCost = X_COST;
    //System.out.println(nurseCost);

    if (TV.equals("V")) { // if (TV.equals("V"))

        TVCost = 40 * (double)days;
        //System.out.println(TVCost);

        if (phone.equals("T"))

            phoneCost = 15 * (double)days;
            //System.out.println(phoneCost);

        else

            phoneCost = X_COST;
            //System.out.println(phoneCost);

    } // if (TV.equals("V"))

    else { // else !(TV.equals("V"))

        TVCost = X_COST;
        //System.out.println(TVCost);

        if (phone.equals("T"))

            phoneCost = 15 * (double)days;
            //System.out.println(phoneCost);

        else

            phoneCost = X_COST;
            //System.out.println(phoneCost);

    }  // else !(TV.equals("V"))

} // else if (room.equals("P"))

else { // else

    roomCost = 310 * (double)days;
    //System.out.println(roomCost);
    nurseCost = X_COST;
    //System.out.println(nurseCost);

    if (TV.equals("V")) { // if (TV.equals("V"))

        TVCost = 40 * (double)days;
        //System.out.println(TVCost);

        if (phone.equals("T"))

            phoneCost = 15 * (double)days;
            //System.out.println(phoneCost);

        else

            phoneCost = X_COST;
            //System.out.println(phoneCost);

    } // if (TV.equals("V"))

    else {  // else !(TV.equals("V"))

        TVCost = X_COST;
        //System.out.println(TVCost);

        if (phone.equals("T"))

            phoneCost = 15 * (double)days;
            //System.out.println(phoneCost);

        else

            phoneCost = X_COST;
            //System.out.println(phoneCost);

    }  // else !(TV.equals("V"))

} // else

Code like the following ...

   if (phone.equals("T"))

        phoneCost = 15 * (double)days;
        //System.out.println(phoneCost);

    else

        phoneCost = X_COST;

... is copy-and-pasted into more than one type of room.

Instead of having a structure like ...

if room type
    calculate nurse cost
    calculate phone cost
    calculate TV cost
else if other room type
    calculate nurse cost again
    calculate phone cost again
    calculate TV cost again

... I suspect it would be better to have a structure like ...

// Calculate TV cost.
TVCost = (room.equals("I") || !TV.equals("V")) ? X_COST : 40 * (double)days;
// Calculate Phone cost
... etc ...
// Calculate nurse cost
... etc ...

Also I don't understand this ...

else if (room.equals("N"))

    nurseCost = 250 * (double)days;
    //System.out.println(nurseCost);

... because that's in a place where you already decided that room.equals("I").


Width-based panel collapsion This code adds the toggle functionality for mobile devices. So, for width lower than 990px, the panel collapses, and the user can toggle to open / close it.

angular.module('app')
.directive('ngContentToggle', function($window) {

    return {
        link: function(scope, $element){
            var w = angular.element($window, $element);

            scope.toggleClass = function(){
                if($window.outerWidth < 990) {
                    $element.addClass('slide-content');
                }
                else {
                    $element.removeClass('slide-content');
                }
            };

            w.bind('resize load', function(){
               scope.toggleClass();
            });

            $element.bind('click', function(){
                if($element.hasClass('slide-content')){
                    if(($element).next().height > 0) {
                        $element.next().toggleClass('slide-content-open');
                    }
                    else {
                        $element.next().toggleClass('slide-content-open');
                    }
                }
            });
        }
    };
});

This won't be a very in-depth review since I don't know angular.js, and I will focus only on some superficial issues.

You have "magic" classes.

These should be declared at the top, to avoid any kind of problem.

An example:

var classes = {
    open: 'slide-content-open',
    slide: 'slide-content'
};

This could help a lot with any future class change.
You repeat $element.next() 3 times.

This shows a little of misuse of the library, and you should be careful with this since it kills performance.

Try to save it into a variable or chain it's methods, if possible.
You have a "magic" number there!

Imagine that now you want to change the width that the class it removed from 990 to, say, 1020. You can argue and say "I just edit it manually, duh!". Now imagine you have 300 lines of code and you don't know where it is and it's previous value.

I recommend storing that value in a variable by itself.
You can eliminate a lot of bloat by reducing duplicated conditions.

You have the following code:

if($element.hasClass('slide-content')){
    if(($element).next().height > 0) {
        $element.next().toggleClass('slide-content-open');
    }
    else {
        $element.next().toggleClass('slide-content-open');
    }
}

So, no matter what you do, it will toggle the class? Why have this whole condition?

Eliminating most of the code:

if($element.hasClass('slide-content')){
    $element.next().toggleClass('slide-content-open');
}

For some reason, this code seems to look a little like jQuery to me. If you have jQuery, this fits in 1 single line:

$element.filter('.slide-content').next().toggleClass('slide-content-open');

You don't use the variable w anywhere, besides in 1 method a few lines below.

Instead of all that, you can chain the methods and spare a dangling reference later on.

Something like this:

angular
    .element($window, $element)
    .bind('resize load', function(){
       scope.toggleClass();
    });

For the final code:

Without jQuery:

angular
    .module('app')
    .directive('ngContentToggle', function($window) {

        var classes = {
            open: 'slide-content-open',
            slide: 'slide-content'
        };

        var minWidth = 990;

        return {
            link: function(scope, $element) {
                angular
                    .element($window, $element)
                    .bind('resize load', function(){
                       scope.toggleClass();
                    });

                scope.toggleClass = function(){
                    if($window.outerWidth < minWidth) {
                        $element.addClass(classes.slide);
                    }
                    else {
                        $element.removeClass(classes.slide);
                    }
                };
                $element.bind('click', function(){
                    if($element.hasClass(classes.slide)){
                        $element.next().toggleClass(classes.open);
                    }
                });
            }
        };
    });

With jQuery:

angular
    .module('app')
    .directive('ngContentToggle', function($window) {

        var classes = {
            open: 'slide-content-open',
            slide: 'slide-content'
        };

        var minWidth = 990;

        return {
            link: function(scope, $element) {
                angular
                    .element($window, $element)
                    .bind('resize load', function(){
                       scope.toggleClass();
                    });

                scope.toggleClass = function(){
                    if($window.outerWidth < minWidth) {
                        $element.addClass(classes.slide);
                    }
                    else {
                        $element.removeClass(classes.slide);
                    }
                };
                $element.bind('click', function(){
                    $element.filter('.' + classes.slide).next().toggleClass(classes.open);
                });
            }
        };
    });


Event sourcing using functional programming Is this a satisfactory implementation of event sourcing using functional programming?

module Account

(*Types*)
type EventAggregate<'Aggregate> = { Event:Event; Aggregate:'Aggregate }

and Command =
    | Login      of EventAggregate<Account>
    | Deposit    of EventAggregate<Account>
    | Withdrawal of EventAggregate<Account>

and Event =
    | LoggedIn  of Login
    | Deposited of decimal
    | Withdrew  of decimal

and AccountId = AccountId of string
and FirstName = FirstName of string
and LastName =  LastName  of string

and Login =   { UserId:string; Password:string }
and Account = {
    AccountId: AccountId
    FirstName: FirstName
    LastName:  LastName }

(*Functions*)
let appendTo (repo:Map<Event,_>) (plan:EventAggregate<'a>) =
    repo.Add (plan.Event , plan.Aggregate)

let getAccount login = {
    AccountId= AccountId "myAccountId"
    FirstName= FirstName "Scott"
    LastName= LastName   "Nimrod" }

let interpret cmd repo = cmd |> function
    | Login   v
    | Deposit v -> appendTo repo v
    | _         -> repo

(*Client*)
let login = { UserId= "MyUserId"; Password= "MyPassword" }
let account = getAccount login

let loginPlan =    { Event=     LoggedIn login
                     Aggregate= account        }

let depositPlan =  { Event=     Deposited 100m
                     Aggregate= account        }

let store =  (Login loginPlan , Map.empty)
             ||> interpret
              |> interpret (Deposit depositPlan)

The key concept in event sourcing is having an 'updater' function that takes an event and an old state, and returns a new state. Let's capture that as a type. The type can be very generic:

module Update =
  type t<'s, 'e> = 's -> 'e -> 's

  (**
  A function that represents the operation of taking an event and an old
  state to return a new state.
  *)

The next thing to capture is your specific domain and its specific implementation of the 'updater' function:

module Account =
  type t = { logged_in : bool; balance : decimal }
  type event =
    | Login of userid:string * passwd:string
    | Deposit of decimal
    | Withdraw of decimal

  let empty = { logged_in = false; balance = 0m }
  let login_valid userid passwd = true // <- TBD
  let update t event =
    match event, t with
      | Login (userid, passwd), _ ->
        { t with logged_in = login_valid userid passwd }

      | Deposit amt, { logged_in = true; balance = balance } ->
        { t with balance = balance + amt }

      | Withdraw amt, { logged_in = true; balance = balance } ->
        { t with balance = balance - amt }

      | _, t -> t

  (* val update : Update.t<t, event>

Finally, you need some stream of events. It can be in the form of some stream data type or an observable sequence. We will use an IObservable<Account.event> because it's idiomatic in the .Net world. With this stream of events, you can 'fold' over the events and regenerate your final state (your account) by applying the update function at each step.

module Main =
  open System.Reactive.Linq

  (* For example only; real stream will be built differently. *)
  let acct_events =
    Observable.Concat(
      [| Observable.Return(Account.Login ("bob", "pass"))
         Observable.Return(Account.Deposit 1000m)
         Observable.Return(Account.Withdraw 500m) |])

  let acct =
    Observable.Aggregate(acct_events, Account.empty, Account.update)

  (** val acct : IObservable<Account.t> *)


Stacks using Linked Lists in C I tried to implement a stack using Linked Lists. How could I make it more efficient? Any other suggestions will be greatly appreciated. I am a beginner C programmer.

Any other tips to clean up my code?

#include <stdio.h>
#include <stdlib.h>

typedef struct NODE {
    int data;
    struct NODE* next;
}node;

node* top = NULL;
// Push Function
void push(int x)
{
    node *temp;
    temp = (node *) malloc(sizeof(node));
    temp -> data = x;
    temp -> next = top;
    top = temp;

}
// Pop function
int pop()
{
    node* temp;
    int num=0;
    temp = top;
    num = temp -> data;
    top = top -> next;
    free(temp);
    return num;
}

int Top()
{
    return top -> data;
}

void display(node *head)
{
    if(head == NULL) printf("NULL!\n");
    else
    {
        printf("%d\n", head -> data);
        display(head->next);
    }
}
int main() {
    int element,choice,val,tp;
    do
    {
    printf("---Stack Operations---\n");
    printf("1.PUSH\n");
    printf("2.POP\n");
    printf("3.DISPLAY\n");
    printf("4.Top element\n");
    printf("5.EXIT\n");
    printf("Enter an option\n");
    scanf("%d",&choice);

    switch(choice)
    {
        case 1:
        printf("Enter the element to push\n");
        scanf("%d",&val);
        push(val);
        break;

        case 2:
        element = pop();
        printf("Popped element is: %d\n",element);
        break;

        case 3:
        display(top);
        break;

        case 4:
        tp = Top();
        printf("Top element is:%d\n",tp);
        break;

        case 5:
        exit(0);
        default:
        printf("Enter a valid option\n");
        }
    }while(choice != 5);

}

Consider alternatives to linked lists

You are wondering whether this is the most efficient way to implement a stack. If you store ints, then the overhead of maintaining a linked list is quite large. Another option would be to keep the integer values in a dynamically allocated array. That way, the only other piece of information you need to store is the size of your stack. The drawback of an array is that if you grow the stack, you have to grow the memory for it, which can be expensive, but if you do this in a smart way the overhead will probably still be less than with linked lists.

You can also go for a hybrid approach: have struct NODE contain a fixed number of ints, say 8 or 16, and only allocate another struct NODE when the previous one is full.

Don't use all-capital names for types

All-capital identifiers are normally used for macros. It's always best to follow conventions. When defining a new struct, I recommend this pattern:

typedef struct foo {
    ...
} foo_t;

Avoid recursion if it is easy to do so

Your function display() cleverly uses recursion to display all the elements in the stack. However, without any optimizations enabled, if your stack is very large, this will actually cause a lot of real stack space to be used! With optimization enabled, the compiler will most likely perform a tail call optimization, but in this case you can easily rewrite the function as a loop:

void display(node *head)
{
    for(node *it = head; it; it = it->next)
        printf("%d\n", it->data);
}

Ensure a pointer is not NULL before dereferencing it

Your functions pop() and Top() blindly dereference the pointer top. If you would call these functions when the stack is empty, your program will crash.

Rename the variable top

This variable is effectively a private variable that you shouldn't access directly, only via the stack manipulation functions that you created (push(), pop() and Top()). Already there is the issue that it has the same name as the function Top(), and while C is case-sensitive, this solution is not pretty. Rename the function Top() to top(), so it matches the other functions, and rename the pointer top to something like stack_top or top_.


Binary to decimal converter I started a few weeks ago to learn C++ online. I just made a little binary → decimal program. Here is my code:

#include <iostream>

using namespace std;

int main()
{
    int i,j=1;
    long long decimal=0, powerOfTwo=1;
    cin>>i;
    char binary[i];
    i*=2;
    while (i)
    {
        if (i > j)
        {
           cin>>binary[j];
           ++j;
        }
        else
        {
            cout<<binary[i];
            if (i%4 == 1) //adds a space after 4 numbers (eg. 1011 0111 instead of 10110111)
                cout<<" ";
            if (binary[i] == '1')
                decimal += powerOfTwo;
            powerOfTwo *= 2;
        }
        --i;
    }
    cout<<"in ten base is equal to "<<decimal;
    return 0;
}

So I want to know, am I heading in the right direction? Please let me know what I am not doing good or any kind of suggestion.

What would Bob do?

You've written an interactive program, but the user does not know what your program does. Therefore, Bob—the generic user—might accidentally write "test" instead of a number, and suddenly your program runs wild.

A first step in the right direction would be to tell the user what your program will do—as long as its interactive. Note that this is not C++ specific.

We can do so by starting with:

cout << "This program converts a binary number to a decimal one\n"
        "Please state the length of your number: ";

But that's error prone. First of all, Bob might not know how long the binary number is. Try to check whether 1101011010101101 is 14 or 15 characters long. That's a trick question, those are 16 characters.

So instead, just let Bob write anything and then check how long it is and whether it was really a binary number.

Prefer std::string to char[N]

Instead of char binary[i], we will use a std::string binary. This makes the input a lot easier. However, to handle the input easier, let's write a function. Actually, let's write two:

bool is_binary_number(const std::string& binary){
    return std::all_of(binary.begin(), binary.end(), [](char c){
        return c == '0' || c == '1';
    });
}

That's using C++11's all_of in combination with a lambda. In case you don't know that yet, here's the same variant written in a range-based for loop:

bool is_binary_number(const std::string& binary){
    for(char c : binary) {
        if(c != '0' && c != '1') {
            return false;
        }
    }
    return true;
}

But this is C++11 syntax. In case you're not familar with C++11 at all, it's almost the same as

bool is_binary_number(const std::string& binary){
    for(size_t i = 0; i < binary.size(); ++i) {
        if(binary[i] != '0' && binary[i] != '1') {
            return false;
        }
    }
    return true;
}

In case you don't know std::string & yet, that's a reference. A reference is an alias to an already existing object. A const reference is an alias where we cannot change the value:

int        value = 15;
int&       ref   = value;
const int& cref  = value;

ref = 12;

std::cout << value << std::endl; // prints 12, since we changed it via ref
std::cout << cref  << std::endl; // prints 12, since we changed it via ref

// does not compile:
cref = 7;           // error; this reference is read only!

The details are a little bit more complicated, but in the end, when we call is_binary_number(on_a_string), we won't make a copy of on_a_string.

Either way, now that we have a function to check whether a string is a binary number, we can write our function to ask Bob for input:

std::string ask_binary_number() {
    std::cout << "Please enter a binary number: ";

    while(true) {
        std::string input;
        std::cin >> input;

        if(is_binary_number(input)) {
            return input;
        } else {
            std::cout << "That wasn't a binary number, try again: ";
        }
    }
}

We can now use this in main:

#include <iostream>
#include <string>        // necessary for std::string

int main () {
    std::string binary = ask_binary_number();

    ...
}

Note that we don't need to touch main at all if we want to change the input method later on. We only need to change ask_binary_number. This makes it easy to ask the user for another number, or re-use ask_binary_number in another project.

Exercise: Usually, you don't want leading zeroes in binary numbers. Which function do you have to change to not allow leading zeroes in the result of ask_binary_number? What changes are necessary?

Keep it simple and stupid

In your for loop, you do two things at once: you print the binary number and convert the number to a decimal. Unless you know that this is a bottleneck in your program and you need cache locality or something similar to keep your code fast, do one thing and one thing only. This makes it a lot easier to change the code later without breaking something else.

We can split this functionality again into two functions:

void print_binary_with_spaces(const std::string & binary, size_t digits = 4){
    // exercise; almost solved by your own code
}

unsigned long binary_to_decimal(const std::string & binary){
    // exercise; almost solved by your own code
}

If you implement all functions given above, you end up with the following main:

#include <iostream>
#include <string>

// functions here (either completely or only their declaration)

int main() {
    std::cout << "This program converts a binary number to a decimal one." << std::endl;
    std::string binary = ask_binary_number();

    std::cout << "Your binary number ";

    print_binary_with_spaces(binary);

    std::cout << " is " << binary_to_decimal(binary) << "in decimal"
              << std::endl;
}

Note that all complex interactions are moved into functions. However, as you have seen in ask_binary_number and is_binary_number those functions aren't complex themselves either. Keeping things simple is a principle (KISS) that's often applied, as is splitting the functionality (separation of concerns).

A concrete example of overly complex code

Why am I pointing to simplicity? Because your while loop is overly complex:

i = i * 2;
j = 1;
while (i)
{
    if (i > j)
    {
       ...            // lets call this A
       ++j;
    }
    else
    {
        ...          // lets call this B
    }
    --i;
}

Essentially, you have the following loops:

for(j = 0; j < i; ++j)
{
    A; // see above
}

for(--i; i > 0; --i)
{
    B; // see above
}

If we used that in your program, it would look like this:

for(int j = 0; j < i; ++j){
    cin >> binary[j];
}

while(i-- > 0){
    cout << binary[i];
    if (i%4 == 1)
        cout << " ";
    if (binary[i] == '1')
        decimal += powerOfTwo;
    powerOfTwo *= 2;
}

This is not only easier to understand, it's even shorter. So try to keep your loops simple.


Checking if a string has all unique characters I'm wondering if this code can be optimized in terms of time complexity. The problem is checking to see if a string has all unique characters.

#include <iostream>
#include <string>
using namespace std;

//Assuming ASCII

bool checkString (string user) {
    if (user.length() > 256)
    {
        return false;
    }

    //creating a boolean array of size 256
    //setting all values to false (0)
    bool check[256] = {false};

    for (int i = 0; i < user.length(); i++)
        {
        int value = user.at(i);
            if (check[value] == true)
            {
                return false;
            }

            else
            {
                check[value] = true;
            }
        }
    return true;
}

int main() {

    string user;
    cout << "Please enter in a string: ";
    cin  >> user;

    cout << checkString(user) << endl;

    return 0;
}

I'm not very good with determining the time complexity. Right now, would it be \$O(n^2)\$ (because of the for loop)?

Looks good to me from a big-O point of view.

C++wise, you have some misconceptions:

//setting all values to false (0)
bool check[256] = {false};

This explicitly initializes only the first element to false; the remaining 255 elements are value-initialized, which in layman's terms means zero-initialized, which means yes they're initialized to false as well in this case; but your comment is misleading, in that

bool uncheck[256] = {true};

would not initialize all 256 elements to true — it would explicitly initialize the first element to true and then value-initialize the other 255 elements (to false).

for (int i = 0; i < user.length(); i++)
    {
    int value = user.at(i);
        if (check[value] == true)
        {
            return false;

This indentation is all screwed up. You should indent whenever you enter a new scope, and dedent when leaving a scope. So for example it would be sensible to write

for (int i = 0; i < user.length(); i++)
{
    int value = user.at(i);
    if (check[value] == true)
    {
        return false;

but actually most programmers would just write

for (int i = 0; i < user.length(); ++i) {
    int value = user[i];
    if (check[value]) {
        return false;

(notice the cuddled braces, the removal of the explicit test-for-equality with true, the default preference for ++i over i++, and the preference for fast operator[] over slow .at() when the index is statically known to be in-bounds).

bool checkString (string user)

This function signature tells the function to take the parameter user by value, which means "make a copy." Since you don't need a copy for any reason, you should by default prefer to take a reference:

bool checkString(const string& user)

(and notice the removal of the extra space in between the function name and its parameter list; we don't write f (x) in C++ for the same reason you don't write “f  (x)” in math class).

It might be an interesting research project for you to figure out what you'd have to change to make // Assuming ASCII no longer required.


Drawing various types of shapes I did the following Excercise:

  simulate single inheritance in C. Let each "base
     class" contain a pointer to an array of pointers to functions (to simulate
     virtual functions as freestanding functions taking a pointer to a "base
     class" object as their first argument). Implement "derivation" by
     making the "base class" the type of the first member of the derived class.

  For each class, initialize the array of "virtual functions" appropriately. To test the ideas, implement a version of "the old Shape example" with the base
  and derived draw() just printing out the name of their class. Use only
  language features and library facilities available in standard C.

I wonder if my approach is good? And as a side question: Is OOP relevant in practice with using c?

Edit: the goal is to simulate c++ like OOP constructs with the limited features of c.

#include <stdio.h>

typedef void(*pfct)(struct Shape*);

typedef struct {
    pfct draw;
}Vtable;

typedef struct{                 // "Base class"
    Vtable* pvtable;
}Shape;

typedef struct{                 // "Derived" from Shape
    Shape shape;
    int radius;
}Circle;

void draw(Shape* this)          // "Virtual function"
{
    (this->pvtable->draw)(this);
}

void draw_shape(Shape* this)
{
    printf("Drawing Shape\n");
}

void draw_circle(Shape* this)
{
    Circle* pcircle = (Circle*)this;
    printf("Drawing Circle with radius = %i\n",pcircle->radius);
}

Shape* init_shape()
{
    Shape* pshape = malloc(sizeof(Shape));
    Vtable* pvtable = malloc(sizeof(Vtable));
    pvtable->draw = &draw_shape;
    pshape->pvtable = pvtable;
    return pshape;
}

Circle* init_circle(int radius)
{
    Circle* pcircle = malloc(sizeof(Circle));
    pcircle->shape = *init_shape();
    pcircle->shape.pvtable->draw = &draw_circle;
    pcircle->radius = radius;
    return pcircle;
}

int main()
{
    Shape* myshape = init_shape();
    Circle* mycircle = init_circle(5);
    draw(myshape);
    draw(mycircle);

    getchar();
    return 0;
}

Simulate single inheritance in c
  I wonder if my approach is good?

Function consistency

I'd expect ..._circle(Shape* this) functions to take Circle* this.  Let the draw() play with the pointer conversions.

void draw(void* this) {
  Shape* pthis = this;
  (pthis->pvtable->draw)(pthis);
}

void draw_circle(Circle* this) {
    printf("Drawing Circle with radius = %i\n",this->radius);
}

// No longer generate a warning
draw(mycircle);

Naming convention

For type Shape, code uses ..._shape, the undefined struct Shape and  Vtable.  I'd expect greater uniformity - perhaps Shape_init(), Shape_draw(), Shape_Vtable, etc.

For Circle, use Circle_draw(), Circle_init(), etc. and not ..._circle().

Avoid text case changes.

Avoid anonymous struct Shape

This problem hints that code is using C++ to compile C code.  Best to use C for C code.

Change typedef void(*pfct)(struct Shape*); to typedef void(*pfct)(Shape*); and move after typedef struct{ ... }Shape;

Missing companion free()

A companion to ..._init(), a ..._uninit() should exist.

Organization

Put Shape code in one place (or file) and Circle in another.

Consider allocating to the object, not the type

It is easier to code right, review and maintain.

// pshape = malloc(sizeof(Shape));
pshape = malloc(sizeof *pshape);

Error checking

Robust code checks memory allocation.

pshape = malloc(sizeof(Shape));
if (pshape == NULL) Handle_OutOfMemory();


Parking places design I got a question:
Design a parking system. There are 3 types of parking spaces: big, medium and small.
Implement ParkingSystem class(int big, int medium, int small). Constructor takes a number of places of types. So ParkingSystem(1,2,3) means there are 1 big place, 2 medium and 3 small.
Add method addCar(int carType) - carType takes one of three values [1,2,3] which represents big car, medium car and small car.
Method should check if there is a place for a car of such size or bigger, if yes then it parks the car in this place and returns true, else false.
For now there is no method to free parking space (when a car is leaving) but it can be added in future.
Example:

ParkingSystem ps = new ParkingSystem(1, 2, 3); // creates parking with 1 big place, 2 medium and 3 small
ps.addCar(2); //true, medium car takes medium place
ps.addCar(2); //true, medium car takes medium place
ps.addCar(2); //true, medium car takes big place
ps.addCar(1); //false, no place for big car

I wrote:
public class ParkingSystem {
    private int big;
    private int medium;
    private int small;

    public ParkingSystem(int big, int medium, int small) {
        this.big = big;
        this.medium = medium;
        this.small = small;
    }

    public boolean addCar(int car) {
        if (car == 1 && isPlaceForBigCar()) {
            return parkBigCar();
        } else if (car == 2 && isPlaceForMediumCar()) {
            return parkMediumCar();
        } else if (car == 3 && isPlaceForSmallCar()) {
            return parkSmallCar();
        }
        return false;
    }

    private boolean parkBigCar() {
        this.big--;
        return true;
    }

    private boolean parkMediumCar() {
        if (medium > 0) {
            this.medium--;
        } else if (big > 0) {
            this.big--;
        }
        return true;
    }

    private boolean parkSmallCar() {
        if (small > 0) {
            this.small--;
        } else if (medium > 0) {
            this.medium--;
        } else if (big > 0) {
            this.big--;
        }
        return true;
    }

    private boolean isPlaceForBigCar() {
        return this.big > 0;
    }

    private boolean isPlaceForMediumCar() {
        return (this.big > 0 || this.medium > 0);
    }

    private boolean isPlaceForSmallCar() {
        return (this.big > 0 || this.medium > 0 || this.small > 0);
    }
}

so it seems to be a little better than the worst solution but I believe there is still a lot to improve. Should I use strategy pattern to park cars or is there even better solution?
Main/Test class:
public class Main {
    public static void main(String[] args) {
        ParkingSystem ps = new ParkingSystem(1, 2, 3); // creates parking with 1 big place, 2 medium and 3 small
        System.out.println(ps.addCar(2)); //true, medium car takes medium place
        System.out.println(ps.addCar(2)); //true, medium car takes medium place
        System.out.println(ps.addCar(2)); //true, medium car takes big place
        System.out.println(ps.addCar(1)); //false, no place for big car
    }
}

this.
There's a lot of this. references in your code.  Typically in Java you only use this. where you need to in order to disambiguate variables.  Consider removing them as they're only adding noise to your code.
always True
Your park methods all return boolean, even though they all always return true. If you want to maintain the same approach, consider changing them to void.
That said, I'd be tempted to merge the isPlaceFor and park logic together into a single tryPark method that checks if it can park and parks it if it can.  This seems like a connected operation.  Are you ever likely to want to know if you can park there without actually parking the car?
Naming / Constants
You fields are big,medium,small.  You use them to track the available spaces of that size, consider being a bit more verbose with the field names to make the meaning clearer.  It'll help when you add maxAvailableBigSpaces for example.
I'd also consider adding constants for the CAR sizes.  Whilst it may be obvious at the moment that 3 is a small car, it may not be as obvious in a week/months time.
delegating
When you park a small car, you're effectively checking for the small space, and if it's not there, trying to park it as a medium car, which if it's not possible you then try to park as a large car.  You could match this approach in your code, which simplifies the logic in each stage and removes some of your duplicate checks.
switch
Combining some of the above together and using a switch expression rather than the if/elseif/else combo means that you could end up with some code more like this:
public class ParkingSystem {
    private static final int BIG_CAR = 1;
    private static final int MEDIUM_CAR = 2;
    private static final int SMALL_CAR = 3;
    private int availableBigSpaces;
    private int availableMediumSpaces;
    private int availableSmallSpaces;

    public ParkingSystem(int bigSpaces, int mediumSpaces, int smallSpaces) {
        availableBigSpaces = bigSpaces;
        availableMediumSpaces = mediumSpaces;
        availableSmallSpaces = smallSpaces;
    }

    public boolean addCar(int car) {
        return switch (car) {
            case BIG_CAR -> tryParkBig();
            case MEDIUM_CAR -> tryParkMedium();
            case SMALL_CAR -> tryParkSmall();
            default -> false;
        };
    }

    private boolean tryParkBig() {
        if(availableBigSpaces > 0) {
            availableBigSpaces--;
            return true;
        }
        return false;
    }

    private boolean tryParkMedium() {
        if (availableMediumSpaces > 0) {
            availableMediumSpaces--;
            return true;
        }
        return tryParkBig();
    }

    private boolean tryParkSmall() {
        if (availableSmallSpaces > 0) {
            availableSmallSpaces--;
            return true;
        }
        return tryParkMedium();
    }
}

A simple array approach
An alternate approach, which is more concise, however I don't think is as clear to follow, would be to use the ordering of the car sizes and an array to handle the allocation.
public class ParkingSystem {
    private int availableSpaces[] = new int[3];

    public ParkingSystem(int bigSpaces, int mediumSpaces, int smallSpaces) {
        availableSpaces[2]=smallSpaces;
        availableSpaces[1]=mediumSpaces;
        availableSpaces[0]=bigSpaces;
    }

    public boolean addCar(int car) {
        if(car > 3) return false;
        for(var spaceToCheck = car-1;spaceToCheck>=0;spaceToCheck--) {
            if(availableSpaces[spaceToCheck] > 0) {
                availableSpaces[spaceToCheck]--;
                return true;
            }
        }
        return false;
    }
}


Codewars "Consecutive strings" kata The challenge description is as follows:

You are given an array strarr of strings and an integer k. Your task is to return the first longest string consisting of k consecutive strings taken in the array.
Example:
longest_consec(["zone", "abigail", "theta", "form", "libe", "zas", "theta", "abigail"], 2)
--> "abigailtheta"

n being the length of the string array, if n = 0 or k > n or k <= 0
return "".

I solved the challenge with as a side goal to follow and use ECMAScript 6 as much as possible. All suggestions on improving the code are welcome!
Note: I did not like using k and n in code, so I used numStr and arrLen respectively.
const longestConsec = (strArr, numStr) => {
  const arrLen = strArr.length
  if (arrLen === 0 || numStr > arrLen || numStr <= 0) {
    return ""
  }
  const consecStrings = getAllConsecutiveStrings(strArr, numStr, arrLen)
  return getFirstLongestString(consecStrings)
}

const getAllConsecutiveStrings = (strArr, numStr, arrLen) => {
  const numConsecStr = arrLen - numStr
  const result = []
  let consecStr
  for (let i = 0; i <= numConsecStr; i++) {
    consecStr = ""
    for (let s = i; s < i + numStr; s++) {
      consecStr += strArr[s]
    }
    result.push(consecStr)
  }
  return result
}

const getFirstLongestString = strArr => {
  let firstlongestString = ""
  let longestLength = 0
  for (let str of strArr) {
    strLen = str.length
    if (strLen > longestLength) {
      firstlongestString = str
      longestLength = strLen
    }
  }
  return firstlongestString
}

Some performance-related nitpicking:

Instead of string concatenation keep an array of string lengths;
Use a running window of current concatenated length: when advancing to the next string simply subtract the first window's element and add the current string length.

Thus the array will be iterated just once, no extra strings interned in the JS engine.

const longestConsec = (strings, count) => {
    let n = strings.length;
    if (!n || count <= 0 || count > strings.length)
        return '';

    let lengths = Array(n);

    // fill the running window
    let windowLen = 0;
    for (let i = 0; i < count - 1; i++) {
        let stringLen = strings[i].length;
        windowLen += stringLen;
        // fill the cache with items 0..count-1, the rest will be filled in the main loop
        lengths[i] = stringLen;
    }

    let maxLen = 0;
    let maxLenAt = 0;
    for (let i = count - 1, windowStart = 0; i < n; i++, windowStart++) {
        let stringLen = strings[i].length;
        lengths[i] = stringLen;

        let thisLen = windowLen + stringLen;
        if (thisLen > maxLen) {
            maxLen = thisLen;
            maxLenAt = windowStart;
        }
        windowLen += stringLen - lengths[windowStart];
    }

    return strings.slice(maxLenAt, maxLenAt + count).join('');
}


Bash script prepares virtual environment I have a few Flask projects that I want to install, each in their own virtual environment. The approach is derived from this tutorial. The steps to create the VE's are very similar, so they're easy to automate. So I wrote this, as a first step. It seems to work OK:

#! /bin/bash

if [ $# -lt 2 ]
  then
    echo -e "mkve.sh - make virtual environment\nUsage: mkve.bash project_name user_name"
    exit
fi

echo project name: $1
echo project user: $2

# create the webapp user and its home directory, and give it ownership
sudo useradd --system --gid webapps --shell /bin/bash --home /mnt/u1/$1 $2
sudo mkdir -p /mnt/u1/$1
sudo chown $2 /mnt/u1/$1

# become the webapp user, create a virtual environment, start it, and install gunicorn and flask
sudo su $2 -c "cd ~; virtualenv ./venv"
sudo su $2 -c "cd ~;. venv/bin/activate; pip install gunicorn flask"

echo -e "Done! Start with:\n$ sudo su - $2\n$ . venv/bin/activate"
echo -e "Or undo the whole kaboodle with:\n$ sudo userdel -r $2"

# at this point, manual configuration is necessary, which is different for each project.

I'm fairly new to bash programming, so I'm open to suggestions.
Is this good practice at all?
Any serious pitfalls I have overlooked?
Any room for improvement?

There are a number of good practice items you are missing here, that would benefit your script.

First up, you should name your variables/parameters. The check you have to ensure that the project name and user are specified is OK, but you should rather pull the $1 and $2 values in to their own variables, say pname and puser.

pname=$1
puser=$2

Then it becomes easier to read the code... $pname is a lot easier to understand than $1.

The second issue I see is that you don't check for error conditions. A relatively easy handler is to add the exit-on-error flag to your shell script. Consider adding set -e at the beginning of the script (or a more elaborate system, but your script should be fine with set -e. For more information see: What does set -e mean in a bash script?

The third issue is a small one - why are you using both sudo and su in the same command..... sudo su .... seems rhetorical. Just use sudo --user $puser ....

Another little issue is that you should exit with a non-zero if the parameters are wrong (use exit 1 instead of exit)...

echo -e "mkve.sh - make virtual environment\nUsage: mkve.bash project_name user_name"
exit

should be:

echo -e "mkve.sh - make virtual environment\nUsage: mkve.bash project_name user_name"
exit 1

Yet another small issue is that it's normal to not have .sh as the extension for exectuable shell scripts. Why call your script mkve.sh instead of mkve? Also, you call it two different things in the error message, mkve.sh and mkve.bash.... which is right?

Finally, it is unconventional to put home directories in the /mnt folder. That folder has a special purpose when mounting file systems that are not normally mounted (things like CD's, USB drives, etc.). Using that folder for these home directories will probably lead to unexpected issues if/when someone mounts a drove over /mnt and all your home folders disappear.

The end code would be something like (untested):

#! /bin/bash

# exit if there's a failed command
set -e

if [ $# -lt 2 ]
  then
    echo -e "mkve - make virtual environment\nUsage: mkve project_name user_name"
    exit 2
fi

pname=$1
puser=$2

echo project name: $pname
echo project user: $puser

# create the webapp user and its home directory, and give it ownership
sudo useradd --system --gid webapps --shell /bin/bash --home /ves/$pname $puser
sudo mkdir -p /mnt/u1/$pname
sudo chown $puser /mnt/u1/$pname

# become the webapp user, create a virtual environment, start it, and install gunicorn and flask
sudo --user $puser bash -c "cd ~; virtualenv ./venv"
sudo --user $puser bash -c "cd ~;. venv/bin/activate; pip install gunicorn flask"

echo -e "Done! Start with:\n$ sudo -u $puser -i\n$ . venv/bin/activate"
echo -e "Or undo the whole kaboodle with:\n$ sudo userdel -r $puser"

# at this point, manual configuration is necessary, which is different for each project.


Efficient substring match I have a list of unique strings (approx 2500000) of different lengths and I am trying to find if there is any string which occurs as a substring of previous strings.

def index_containing_substring(the_list, substring):
    for i, s in enumerate(the_list):
        if substring in s:
            return i
    return -1

def string_match():
    test_list=['foo bar abc xml','fdff gdnfgf gdkgf','foo bar','abc','xml','xyz']
    max_len=4 # I am storing the maximum length of sentence
    # the list starts with reverse order
    # i.e sentence with highest length are at the top
    safe_to_add=[]
    for s in test_list:
        if len(s)==max_len:
            safe_to_add.append(s)
        else:
            idx=index_containing_substring(safe_to_add,s)
            if idx==-1:
                safe_to_add.append(s)
            else:
                # process the substring
                print('match found <{}> for <{}>'.format(test_list[idx],s))

This method works fine but I think it is pretty slow. Is there a better way to solve this problem using a better data structure (trie or suffix tree)?

Output

match found <foo bar abc xml> for <foo bar>
match found <foo bar abc xml> for <abc>
match found <foo bar abc xml> for <xml>

This is one of those times when "the rubber meets the road" as far as big-O complexity goes. You've got n = 2.5e6 words, so all that stuff about how \$O(\log n) < O(n) < O(n\log n) < O(n^2)\$ starts to be important.

Existing performance

You'll want to get a meaningful set of sample data, maybe 10 or 100 thousand words, and you'll want to look at using the timeit module, or something else you devise, to measure real performance.

That said, let's look at your code:

def index_containing_substring(the_list, substring):
    for i, s in enumerate(the_list):
        if substring in s:
            return i
    return -1

This function is called with a single word from the input list of words. It checks that one word against all the words already present in the_list. To do that, it iterates over the_list once, and then uses the in operator on each value.

Eventually, the_list is going to be sized based on the input word list. If nearly every word in the input list is a substring of some larger word, for example: [ 'aaaa', 'aaa', 'aa', 'a' ] then you will have one member. But if every word in the input is distinct, for example: [ 'a', 'b', 'c', 'd' ] you will have len(the_list) == N where N is the number of input words. For the worst case, we have to assume this.

For the use of substring in s, Python's substr in string is worst-case \$O(s \cdot t)\$, according to this post. In this analysis, s and t are lengths of the string and substring. Since your substrings eventually get added to the list, s == t. Thus, your index_containing_substring is worst-case \$O(n \cdot s^2)\$ (and average case about \$O(n \cdot s)\$).

def string_match():
    test_list=[...]
    safe_to_add=[]

    for s in test_list:
        if len(s)==max_len:
            safe_to_add.append(s)
        else:
            idx=index_containing_substring(safe_to_add,s)
            if idx==-1:
                safe_to_add.append(s)
            else:
                ...

In this function, you iterate once over the entire list of strings, calling the index_containing_substring function for almost every one. Your behavior is worst-case \$O(n)\$ where n is the length of the word list, times the behavior of the index_containing_substring function, which is \$O(n \cdot s^2)\$. Thus, your overall performance is \$O(n^2 \cdot s^2)\$, which since \$n \gg s\$ simplifies to \$O(n^2)\$.

It's worth noting that there are some "constant factors" in there which get ignored. For example, every time through the loop you check the length of your inputs to determine if you can add them without checking if they are contained substrings. This is a mistake for two reasons: first, what if they are duplicates? I don't see you doing any quality checks on the data, so I can't guarantee that two copies of "the longest word in the list" don't appear, and get added twice. Second, of course, is that you say the words are sorted in reverse order by length. This means that doing the substring-check on the first, long words is the cheapest operation possible. Checking for substrings in a list of length 0, or 1, is much much cheaper than it will ever be again. So you aren't saving very much. But you are doing that length comparison every single time.

I'd suggest that you either (1) abandon all that stuff entirely, and just pay the small price for the first couple of words; or (2) create a separate loop at the beginning to filter those words into the list, and then run your main loop starting after those are done.

An alternative

Suppose you computed every substring of a word. That would require iterating across the entire word, and for each point you reached you would iterate across all possible lengths of the substring:

word, wor, wo, w
 ord, or, o
  rd, r
   d

That is going to be \$O(s^2)\$ where s is the length of your word.

Now suppose you added all of the generated substrings to a Python set     that has \$O(1)\$ access time. You could do something like this:

for word in big_list:
    if word in set_of_all_substrings:
        ...
    else:
        add_all_substrings_of(word, set_of_all_substrings)

Assuming your "found" condition is \$O(1)\$, the performance is \$O(n \cdot s^2)\$, which again simplifies to \$O(n)\$.

Another alternative

Suppose you build a table of suffixes of all the words. That is, word[i:] for i in range(len(word)). Adding a single word's suffixes would be \$O(s)\$ as you can see, instead of \$O(s^2)\$ for substrings.

You could then binary-search the list, comparing word < suffix[:len(word)] to test for membership. The binary search would be \$O(\log n)\$ and you would do it n times, so your overall performance would be \$O(n \log n)\$. This seems worse than the \$O(n)\$ above, but it might be worth checking. The log(2.5 million) is about 22, and if your words have an average length of 10 letters, the difference between \$s\$ and \$s^2\$ would be 10, putting the two techniques a factor of 2.2 apart in performance. That's the kind of difference that code being written in C versus Python, or being particularly well-tuned for a particular purpose, can overcome.

Note that I don't expect this to work: inserting values into a sorted list gets quite expensive, so I suspect you'll find that this code loses the battle of small performance tweaks. But that's why they run horse races, eh?


Web-scraping through a rotating proxy script I've created a script in python which is able to parse proxies (supposed to support "https") from a website. Then the script will use those proxies randomly to parse the title of different coffe shops from a website. With every new request, the script is supposed to use new proxies. I've tried my best to make it flawless. The scraper is doing fine at this moment.

I'll be happy to shake off any redundancy within my script (I meant DRY) or to bring about any change to make it better.

This is the complete approach:

import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
from random import choice

links = ['https://www.yellowpages.com/search?search_terms=Coffee%20Shops&geo_location_terms=Los%20Angeles%2C%20CA&page={}'.format(page) for page in range(1,6)]

def get_proxies():
    link = 'https://www.sslproxies.org/'
    response = requests.get(link)
    soup = BeautifulSoup(response.text,"lxml")
    proxies = [':'.join([item.select_one("td").text,item.select_one("td:nth-of-type(2)").text]) for item in soup.select("table.table tr") if "yes" in item.text]
    return proxies  #producing list of proxies that supports "https"

def check_proxy(session, proxy_list=get_proxies(), validated=False):
    proxy = choice(proxy_list)
    session.proxies = {'https': 'https://{}'.format(proxy)}
    try:
        print(session.get('https://httpbin.org/ip').json())
        validated = True  #try to make sure it is a working proxy
        return
    except Exception: pass

    while True:
        proxy = choice(proxy_list)
        session.proxies = {'https': 'https://{}'.format(proxy)}
        if not validated:  #otherwise get back to ensure it does fetch a working proxy
            print("-------go validate--------")
            return

def parse_content(url):
    ua = UserAgent()
    session = requests.Session()
    session.headers = {'User-Agent': ua.random}
    check_proxy(session)  #collect a working proxy to be used to fetch a valid response

    while True:
        try:
            response = session.get(url)
            break     #as soon as it fetch a valid response, it will break out of the while loop to continue with the rest
        except Exception as e:
            session.headers = {'User-Agent': ua.random}
            check_proxy(session)  #if exception is raised, start over again
            parse_content(url)

    soup = BeautifulSoup(response.text, 'lxml')
    for items in soup.select(".info span[itemprop='name']"):
        print(items.text)

if __name__ == '__main__':
    for link in links:
        parse_content(link)

From your description you want your code to perform these tasks:

Get a list of proxies
That support https
That are actually working

And you want that list to be randomized (from your description without repetition, from your code repetitions are fine).

I would use a couple of generators for that:

import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
from random import shuffle

def get_proxies(link):
    response = requests.get(link)
    soup = BeautifulSoup(response.text,"lxml")
    https_proxies = filter(lambda item: "yes" in item.text,
                           soup.select("table.table tr"))
    for item in https_proxies:
        yield "{}:{}".format(item.select_one("td").text,
                             item.select_one("td:nth-of-type(2)").text)

def get_random_proxies_iter():
    proxies = list(get_proxies('https://www.sslproxies.org/'))
    shuffle(proxies)
    return iter(proxies)  # iter so we can call next on it to get the next proxy

def get_proxy(session, proxies, validated=False):
    session.proxies = {'https': 'https://{}'.format(next(proxies))}
    if validated:
        while True:
            try:
                return session.get('https://httpbin.org/ip').json()
            except Exception:
                session.proxies = {'https': 'https://{}'.format(next(proxies))}

def get_response(url):
    session = requests.Session()
    ua = UserAgent()
    proxies = get_random_proxies_iter()
    while True:
        try:
            session.headers = {'User-Agent': ua.random}
            print(get_proxy(session, proxies, validated=True))  #collect a working proxy to be used to fetch a valid response
            return session.get(url) # as soon as it fetches a valid response, it will break out of the while loop
        except StopIteration:
            raise  # No more proxies left to try
        except Exception:
            pass  # Other errors: try again

def parse_content(url):
    response = get_response(url)
    soup = BeautifulSoup(response.text, 'lxml')
    for items in soup.select(".info span[itemprop='name']"):
        print(items.text)

if __name__ == '__main__':
    url = 'https://www.yellowpages.com/search?search_terms=Coffee%20Shops&geo_location_terms=Los%20Angeles%2C%20CA&page={}'
    links = [url.format(page) for page in range(1, 6)]
    for link in links:
        parse_content(link)

This actually makes sure that no proxy is reused for each site. The order in which the proxies are tried is different for each site. If you are not fine with trying the same proxies again for a new site, just call get_random_proxies_iter outside of parse_content and feed it all the way down to get_proxy.


Playing with event-sourcing I'm playing around with some ideas about how to read aggregates from events.

From a high-level view I would love to have this:

class Apply ev ag where
  apply :: ag -> ev -> ag

which to define how an Event ev should change an Aggregate ag if there is an instance of this class for the pair (if there is none then the aggregate should ignore the event).

If you look at the code below it's not hard at all to do it (please forgive me the anti-pattern with the Event list there).

My problem starts when I want the events all serialized - or better: once I want them deserialized and applied again (to be honest: I don't care about the representations at all - but it should be possible to fold an aggregate from a heterogeneous list/stream of serialized events of course).

What I don't like about my solution is this ugly bit of boiler-plate:

instance DeserializableFor Person where
  tryDeserialize (tn, sn)
    | tn == typeOf (SetName "") = Just (App (read sn :: SetName))
    | tn == typeOf (SetAge 0) = Just (App (read sn :: SetAge))
    | otherwise = Nothing

which could probably be generated using TH but I would prefer a direct solution.

Any suggestions?

{-# LANGUAGE MultiParamTypeClasses, ExistentialQuantification #-}
{-# LANGUAGE DeriveDataTypeable #-}

module Test where

import Data.Data
import Data.Maybe (catMaybes)

class Apply ev ag where
  apply :: ag -> ev -> ag

data Applicable a = forall e . (Show e, Read e, Typeable e, Apply e a) => App e

type Events a = [Applicable a]

fold :: Events a -> a -> a
fold [] a = a
fold ((App e):es) a = fold es $ apply a e

-----------------------------------------
-- nasty solution

serialize :: Events a -> [(TypeRep, String)]
serialize = map (\ (App e) -> (typeOf e, show e))

-- is there a way to get something like this from the `Apply` contexts?
class DeserializableFor a where
  tryDeserialize :: (TypeRep, String) -> Maybe (Applicable a)

-- because THIS is really ugly
instance DeserializableFor Person where
  tryDeserialize (tn, sn)
    | tn == typeOf (SetName "") = Just (App (read sn :: SetName))
    | tn == typeOf (SetAge 0) = Just (App (read sn :: SetAge))
    | otherwise = Nothing

deserialize :: DeserializableFor a => [(TypeRep, String)] -> Events a
deserialize = catMaybes . map tryDeserialize

--------------------------------------------------------
-- Example

data SetName = SetName String
  deriving (Show, Read, Typeable)

data SetAge = SetAge Int
  deriving (Show, Read, Typeable)

data SomeOtherEvent = SomeOtherEvent
  deriving (Show, Read, Typeable)

data Person = Person { name :: String, age :: Int }
  deriving Show

nobody :: Person
nobody = Person "" 0

instance Apply SetName Person where
  apply p (SetName n) = p { name = n }

instance Apply SetAge Person where
  apply p (SetAge a) = p { age = a }

events :: Events Person
events = [App (SetName "Mr.Wrong"), App (SetAge 42)]

serialized :: [(TypeRep, String)]
serialized = (typeOf SomeOtherEvent, show SomeOtherEvent) : serialize events

person :: Person -- is `Person {name = "Mr.Wrong", age = 42}`
person = fold (deserialize serialized) nobody

Below is my attempt. I wrote generic serialization using Generics and cereal. So all that's needed is to derive Generic and add empty Serialize instances. Function save first writes the type of the value, which is then checked by load. So when loading a list of events, we can check wich ones belong to the given type we want to change and drop all others (deserializeMany).

I dropped Applicable completely. Usually the problem with existential data types is that there is only a limited set of things you can do with them, so you can replace such a data type directly with the action, like a function that acts on a given object ag -> ag. See also my question Why (or why not) are existential types considered bad practice in functional programming?.

I also changed the open class of events for a given type (open in the sense any module can define additional events for an object) to a closed one - every WithEvent instance has exactly one data type that describes its possible events. This makes it somewhat easier to handle, in particular avoids some cumbersome situations where it's necessary to explicitly give types when applying events, and it's also less verbose.

{-# LANGUAGE DeriveDataTypeable, DeriveGeneric, ScopedTypeVariables,
             TypeFamilies, FlexibleContexts #-}
module Test where

import Control.Monad
import qualified Data.ByteString as BS
import Data.Maybe (catMaybes)
import Data.Typeable
import Data.Serialize
import Data.Maybe (catMaybes)
import GHC.Generics

-- * Generic serialization of events

-- ** Generic serialization of classes that know their type

save :: (Serialize a, Typeable a) => a -> Put
save x = put (show $ typeOf x) >> put x

-- | Returns 'Nothing' if the serialized event is of a different type.
-- Note that that's different from parsing error.
load :: forall a . (Serialize a, Typeable a) => Get (Maybe a)
load = do
    t <- get
    if t == show (typeRep (Proxy :: Proxy a))
      then Just <$> get
      else return Nothing -- some other type

serialize :: (Serialize a, Typeable a) => a -> BS.ByteString
serialize = runPut . save

-- | Read given events and returns the ones that apply to @a@.
deserializeMany :: (Serialize a, Typeable a) => [BS.ByteString] -> Either String [a]
deserializeMany = liftM catMaybes . mapM (runGet load)

-- ** WithEvent

class (Serialize (Event a), Typeable (Event a)) => WithEvent a where
    type family Event a
    apply :: Event a -> a -> a

fold :: (WithEvent a) => [Event a] -> a -> a
fold = flip (foldr apply)

-- * Example

-- ** Foo

data Foo = Foo

data SomeOtherEvent = SomeOtherEvent
  deriving (Show, Read, Typeable, Generic)

instance Serialize SomeOtherEvent where

instance WithEvent Foo where
    type Event Foo = SomeOtherEvent
    apply SomeOtherEvent Foo = Foo

-- ** Person

data Person = Person { name :: String, age :: Int }
  deriving Show

data PersonEvent
    = SetName String
    | SetAge Int
  deriving (Show, Read, Typeable, Generic)

instance Serialize PersonEvent where

instance WithEvent Person where
    type Event Person = PersonEvent
    apply (SetName n) p = p { name = n }
    apply (SetAge a) p = p { age = a }

-- ** Tests

nobody :: Person
nobody = Person "" 0

events :: [PersonEvent]
events = [SetName "Mr.Wrong", SetAge 42]

serialized :: [BS.ByteString]
serialized = serialize SomeOtherEvent : map serialize events

person :: Either String Person -- is `Person {name = "Mr.Wrong", age = 42}`
person = flip fold nobody <$> deserializeMany serialized


JavaScript on page load and on scroll functions Would it serve much purpose to work at refactoring the latter parts of this code?

// This part is OK
function isScrolledIntoView(elem, trigger_height) {

  var docViewTop = $(window).scrollTop();
  var docViewBottom = docViewTop + $(window).height();
  var elemTop = elem.offset().top;

  if (trigger_height == false) {
    trigger_height = elem.height()
  }
  var elemBottom = elemTop + trigger_height;

  return ((elemBottom <= docViewBottom) && (elemTop >= docViewTop));
}

// This function happens when the page first loads
$("p").addClass('hidden').each(function() {
  $this = $(this);
  if (isScrolledIntoView($this, false)) {
    $this.removeClass("hidden");
  }
});

// This function happens on scroll, but basically repeats itself
$(window).off('scroll').scroll(function() {
  $("p.hidden").each(function() {
    $this = $(this);
    if (isScrolledIntoView($this, false)) {
      $this.removeClass("hidden")
    }
  });
});

To me, it seems like there isn't – but at the same time its repetitive:

$this = $(this);
if (isScrolledIntoView($this, false)) {
  $this.removeClass("hidden")
}

Is there a general rule that anyone follows such as "only refactor if you're looking at > 5 lines of repeated code", or should you be refactoring absolutely everything as you go?

I can't really see much to refactor in your code except a few things, so I'll go through those.

if (trigger_height == false)

could be if (!(trigger_height))

$this = $(this);
if (isScrolledIntoView($this, false)) {
    $this.removeClass("hidden");
}

I don't really see why you need to initialise $this, you could just use $(this) later on instead.


Implementation of a threshold detection function in Python I want to implement following trigger function in Python:

Input:

time vector t [n dimensional numpy vector]
data vector y [n dimensional numpy vector] (values correspond to t vector)
threshold tr [float]
Threshold type vector tr_type [m dimensional list of int values]

Output:

Threshold time vector tr_time [m dimensional list of float values]

Function:

I would like to return tr_time which consists of the exact (preffered also interpolated which is not yet in code below) time values at which y is crossing tr (crossing means going from less then to greater then or the other way around). The different values in tr_time correspond to the tr_type vector: the elements of tr_type indicate the number of the crossing and if this is an upgoing or a downgoing crossing. For example 1 means first time y goes from less then tr to greater than tr, -3 means the third time y goes from greater then tr to less then tr (third time means along the time vector t)

For the moment I have next code:

import numpy as np
import matplotlib.pyplot as plt

def trigger(t, y, tr, tr_type):
    triggermarker = np.diff(1 * (y > tr))
    positiveindices = [i for i, x in enumerate(triggermarker) if x == 1]
    negativeindices = [i for i, x in enumerate(triggermarker) if x == -1]
    triggertime = []
    for i in tr_type:
        if i >= 0:
            triggertime.append(t[positiveindices[i - 1]])
        elif i < 0:
            triggertime.append(t[negativeindices[i - 1]])
    return triggertime

t = np.linspace(0, 20, 1000)
y = np.sin(t)
tr = 0.5
tr_type = [1, 2, -2]
print(trigger(t, y, tr, tr_type))
plt.plot(t, y)
plt.grid()

Now I'm pretty new to Python so I was wondering if there is a more Pythonic and more efficient way to implement this. For example whitout for loops or without the need to write seperate code for upgoing or downgoing crossings.

thanks!

If y will always be a sine wave, then you're going about this the wrong way. Rather than calculating trigger_marker based on whether y exceeds tr, you should be calling asin(tr) to see where in the cycle the trigger will be crossed. You would then avoid the need for most of your lists and loops.


Forcing array of promises to complete before returning I'm making one call to a mongoDB database to fetch a list of existing collections and then looping through the collection list and fetching their document count.

All db methods are promises and i'm getting them to fully resolve before responding to an API req. Here's my code

let records = {}
let promises = []

nosqldb.get().listCollections().toArray()
  .then(function(collections) {
    collections.map((coll) => {
      var collectionName = coll.name;
      var collection = nosqldb.get().collection(collectionName);
      promises.push(collection.count().then((count) => {
        records[collectionName] = count;
      }));
    });
    return Promise.all(promises);
  })
  .then(function() {
    return res.status(200).json(records);
  })
  .catch(function(err) {
    return res.status(500).send(err);
  });

As you can see , i'm looping through all collection names and asking the db to return their document count. I push these promises to an Promise array and having all of them resolve before sending the response.

Is this approach correct or is there a more elegant way?

Thanks

forEach vs map

When using map function, we're supposed to return something back from the function parameter. In the code above you do map and populate the promises collection by pushing to it. If you want to have the external collection as it is now, a forEach should be more readable than a map.

With map, the function may look like this. Also, notice that this allows joining the declaration and the assignment of promises array within the body of then(...):

let records = {}
nosqldb.get().listCollections().toArray()
  .then(collectionList => {
    const promises = collectionList.map(collection =>
      nosqldb.get().collection(collection.name).count().then(count => { records[collection.name] = count; })
    );
    return Promise.all(promises);
  })
  .then(() => res.status(200).json(records))
  .catch((err) => res.status(500).send(err));

Side note

You may want to limit the number of collections in which you do .count(). If you have dozens or hundreds or even more collections, you this function may attempt to open as many connections (IDK the details). Also, the function execution time will be really long.

A typical solution in limiting is pagination and/or filtration of the things being counted.


how to simplify this code I'm just wondering if you see any ways I could simplify the following code without making it too difficult for a newbie to understand what I'm trying to do:

    if (trim($this->uri->segment(5)) == 1) {

    if ( $this->my_model->functionA($this->uri->segment(8),$this->uri->segment(4)) )
    {
        $data['light'] =  $this->my_model->data();
    }
    else
    {
        show_error($this->my_model->errormessage());
    }
    }
    elseif (trim($this->uri->segment(5)) == 0) {

      if ( $this->my_model->functionB($this->uri->segment(8),$this->uri->segment(4)) )
      {
          $data['light'] = $this->my_model->data();
      }
      else
      {
          show_error($this->my_model->errormessage());
      }
    }
    elseif (trim($this->uri->segment(5)) == 2) {

      if ( $this->my_model->functionC($this->uri->segment(8),$this->uri->segment(4)) )
      {

          $data['light'] = $this->my_model->data();
      }
      else
      {
          show_error($this->my_model->errormessage());
      }
    }

I guess I could use a switch statement but i would still need to check the results of each function call.  Would it still be beneficial to use switch?  if so, can you explain why?

Any suggestions would be appreciated.
thanks.

I haven't been able to test it, but I'm fairly sure this will work:

$functionName = 'function';
switch(trim($this->uri->segment(5)))
{
    case 1: $functionName .= 'A'; break;
    case 0: $functionName .= 'B'; break;
    case 2: $functionName .= 'C'; break;
}
if($this->my_model->$functionName($this->uri->segment(8), $this->uri->segment(4)))
{
    $data['light'] = $this->my_model->data();
}
else
{
    show_error($this->my_model->errormessage());
}

For more information, see the PHP page on Variable Functions.


How can I improve this code without using static or singleton? I used to use static or singleton for my DAL class. However, I read some articles saying that singleton is evil and should be avoided. Therefore I try to rewrite my code like this:

public class CustomerBLL
{
    private CustomerHelper helper = new CustomerHelper();
    private CustomerDAL dal = new CustomerDAL();

    public void Method1(string param)
    {
        dal.Method1(param);
    }

    public void Method2(string param)
    {
        dal.Method2(param);
    }

    public void MethodA(int param)
    {
        helper.MethodA(param);
    }
}

How do I improve this code?  It just doesn't look right.

I believe that some generalizations are to be made lightly. i.e. stating that Singletons are evil in general is not accurate, as there are some cases where Singleton is a proper (and maybe the only) solution. And yes I do believe that it is one of the overly abused  patterns out there. Having gotten that out of the way, I move to answer your question.

From the code sample you submitted, it is very clear that your CustomerBLL has no business logic and merely acts as a Façade on top of the data access and other helper classes. Facades should be stateless by nature.

Having said that, I will offer two choices based on testability and decoupling requirements.

If you want to make you CustomerBLL testable and decouple it from the CustomerHelper and CustomerDAL concretions and substitute them with abstractions, then one good design enhancement would be to inject the CustomerHelper and CustomerDAL dependencies into the CustomerBLL class, which allows you to mock and stub the data access and helper classes and focus on testing other code modules that consume the CustomerBLL class. You acheive that through extraxcting interfaces of course.

Here is an example of how the code would look if we inject the dependencies through a constructor, note that your CustomerBLL Façade will be an instance class i.e. no static methods will be exposed.

public class CustomerBLL
{
    private readonly ICustomerHelper _helper;
    private readonly ICustomerDAL _dal;

    // default constructor initializes the concrete dependencies
    public CustomerBLL()
        :this(new CustomerHelper(), new CustomerDAL())
    {}

    // constructor to inject stubbed dependencies for testing purposes
    public CustomerBLL(ICustomerHelper customerHelper, ICustomerDAL customerDal)
    {
        _helper = customerHelper;
        _dal = customerDal;
    }

    public void Method1(string param)
    {
        _dal.Method1(param);
    }

    public void Method2(string param)
    {
        _dal.Method2(param);
    }

    public void MethodA(int param)
    {
        _helper.MethodA(param);
    }
}

public interface ICustomerDAL
{
    void Method1(string s);
    void Method2(string s);
}

public interface ICustomerHelper
{
    void MethodA(int i);
}

internal class CustomerDAL : ICustomerDAL
{
    public void Method1(string s)
    {
    }

    public void Method2(string s)
    {
    }
}

internal class CustomerHelper : ICustomerHelper
{
    public void MethodA(int i)
    {
    }
}

If on the other hand, testability is not a concern at all, then you can proceed by consuming the CustomerBLL as a Façade in a procedural way through static methods (although I much prefer the testability approach, but it’s not every one’s cup of tea).

Here you notice I removed the private fields that hold references to the Helper and DAL classes, because we want to promote that the CustomerBLL Façade is stateless.

public class CustomerFacade
{
    public static void Method1(string param)
    {
        new CustomerDAL().Method1(param);
    }

    public static void Method2(string param)
    {
        new CustomerDAL().Method2(param);
    }

    public static void MethodA(int param)
    {
        new CustomerHelper().MethodA(param);
    }
}

I have seen and used both approaches, in my humble opinion they both are valid depending on the usage scenario. In general I am in favor of testable design and decoupled modules and that’s why I proposed the first approach.


URL checker is very slow I have +-645 IRL links I need to validate. The code below is my attempt, but my problem is that it takes forever to check all links and sometimes times out. Is there a better way of validating so many links?

index.php

<?php
error_reporting(E_ALL);
ini_set('display_errors', 1);

$notValid = [];

// This array contains +-645 url in this array, http://www.example.com is just for example purposes.
$companyData = ['http://www.example.com','http://www.example.com', 'http://www.example.com', 'http://www.example.com'];
// Basic structure of $companyData array

Array
(
    [0] => Array
        (
            [Name] => Name-Ltd
            [Code] => XYZ
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd
        )

    [1] => Array
        (
            [[Name] => Name-Ltd
            [Code] => ZXY
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd
        )

    [2] => Array
        (
            [Name] => Name-Ltd
            [Code] => ZYX
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd
        )

    [3] => Array
        (
            [Name] => Name-Ltd
            [Code] => YZX
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd
        )

    [4] => Array
        (
            [Name] => Name-Ltd
            [Code] => XZY
            [Url] => http://www.example.co.za/example-click-a-company/Name-Ltd

)

// This is to loop through all url's
for ($i=0; $i < count($companyData); $i++)
{
    $url = $companyData[$i]['Url'];

    $headers = get_headers($url, 1);

    if(@$headers['Content-Length'] == 1153)
    {
        $exists = 'not valid';

        $notValid[$i] = $companyData[$i]['Url'];
    }
    else
    {
        $exists = 'valid';
    }
}

foreach ($notValid as $not)
{
    echo "The following URL's need your attention: <br>";
    echo $not;
}

?>

Your script is slow because you are doing a sequence of HTTP REQUESTS. Most of the time your server / script is just waiting for the RESPONSE of the requested URL.

small things you can do

as mentioned yourself in the comments you can change your max execution time to prevent reaching it. (which does not make your script any faster ; -)
move away from get headers and use curl or similar to have the ability to change the connection timeout. You could set it to 1 sec and flag every url which fails to manual review (assuming all healthy urls will respond in time , and some slow false positives). Small benefit: your script should be finished in max url*sec time

big things

so the problem is: in practice is your script waiting for responses, in theory it does not have to , as the checking of an url does not depend on each other. Classic case for parallelization. Downside: php is not the language you want to do parallelization in.

upside: you are not the first with this problem

https://stackoverflow.com/questions/9308779/php-parallel-curl-requests
http://reactphp.org/

or use golang or something with build in parallelization


String manipulations (reverse a string by characters, permutation) I'm trying to solve simple Java problems in various ways. Could anyone comment about the code, like how it could be better, and what could be a problem?

// When using already-defined-method is allowed
static String reverseStringByChar1(String str) {
    return str == null ? null : new StringBuilder(str).reverse().toString();
}

// When using already-defined-method is not allowed
static String reverseStringByChar2(String str) {
    if (str == null)
        return null;

    int length = str.length();
    char[] charArray = new char[length];

    for (int i = length-1, j = 0; i >= 0; i--, j++)
        charArray[i] = str.charAt(j);

    return new String(charArray);
}

// When using extra data structure is not allowed
// (Is there any way not to use extra memory? Here like the temp.)
static String reverseCharacterArray(char[] chars) {
    if(chars.length == 0)
        return null;

    for (int i = 0, j = chars.length-1; i < chars.length / 2; i++, j--) {
        /* char temp = chars[i];
        chars[i] = chars[j];
        chars[j] = temp;
        */
        char temp = chars[i];
        chars[i] = chars[chars.length-1-i];
        chars[chars.length-1-i] = temp;

    }
    return new String(chars);
}

// When comparing two strings to see if they are permutations of each other.
static boolean arePermutations(String str1, String str2) {
    // can be trimmed
    // need to know what to do with uppercases and lowercase

    if (str1 == null || str2 == null || str1.length() != str2.length())
        return false;

    // assuming askii code
    boolean[] str1Array = new boolean[128];

    for (char c : str1.toCharArray())
        str1Array[c] = true;

    for (char c : str2.toCharArray()) {
        if (!str1Array[c])
            return false;
    }

    return true;
}

There's a problem with the algorithm for arePermutuations. For instance, arePermutations("aab", "bba") will return true.


Implementation of a managed wrapper DLL I'm not entirely sure on how to handle errors on this:

Is returning a nullptr in case of bad parameters okay, or is it better to return a ENFMPCodegenData.Empty, like there is Rectangle.Empty or Point.Empty in the .NET Framework?
What about handling errors, if any should occur when creating the native object Codegen?
Should I just encompass this in a try / catch block and wrap a native exception inside  a managed one?

This my first attempt on creating such helper.  It's working, but I'm not really sure about its robustness.

public ref class ENMFPCodegen
{
    private:
        ENMFPCodegen()
        {
        }

    public:

        static ENMFPCodegenData ^ GetData ( array<float> ^ pcm , uint numberOfSamples, uint start_offset )
        {
            if ( pcm == nullptr || pcm->Length <= 0  || numberOfSamples <= 0 )
                return nullptr;

            pin_ptr<float> pFloat = &pcm[0];
            Codegen * codegen = new Codegen ( pFloat, numberOfSamples, start_offset );

            if ( codegen == nullptr )
                return nullptr;

            std::string codeString = codegen->getCodeString();
            const char* chars = codeString.c_str();
            String ^ code = gcnew System::String ( chars );
            int numCodes =  codegen->getNumCodes();
            float version = codegen->getVersion();
            delete codegen;
            ENMFPCodegenData ^ data = gcnew ENMFPCodegenData ( code, numCodes, version );
            return data;
        }

};

public ref class ENMFPCodegenData {
    internal:
        ENMFPCodegenData ( String ^ code, int numCodes, float version )
        {
            Code = code;
            NumCodes = numCodes;
            Version = version;
        }

    public:
        String ^ Code;
        int NumCodes;
        float Version;

        virtual String ^ ToString() override
        {
            StringBuilder ^ sb = gcnew StringBuilder;
            sb->AppendFormat ( "{{Version: {0}, NumCodes: {1}, Code: {2}}}", Version, NumCodes, Code );
            return sb->ToString();
        }
};

Is returning a nullptr in case of bad parameters okay?

No, you should instead throw an ArgumentException (or possibly one of its derived types). This way, if an user of your code makes a mistake, they will get an exception clearly explaining the problem, instead of getting a confusing NullReferenceException at a later point.

  What about handling error if any should occur when creating the native object Codegen?

Again, you should throw a suitable .Net exception. In this case, I think a custom exception would be appropriate.

  Should I just encompass this in a try/catch block and wrap a native exception inside a managed one?

Not necessarily. Unmanaged exceptions are represented as the managed SEHException. So, if that's enough for you, you don't have to do anything.

But if you want to be able to handle some specific unmanaged exceptions from the managed code, or if you want to preserve some information from the unmanaged exception (SEHException doesn't really do that), you should catch the unmanaged exception and throw a new managed exception. For example:

public ref class UnmanagedException : Exception
{
public:
    UnmanagedException(String^ message)
        : Exception(message)
    {
    }
};

…

try
{
    // call some code that can throw std::exception here
}
catch (const std::exception &ex)
{
    throw gcnew UnmanagedException(gcnew String(ex.what()));
}


Simulating motion of particles Description

The code does particle simulation. In this code, some particles take steps with a certain distribution length. if particles go beyond a curve, the code put them back.

Code

   #include <iostream>
#include <vector>
#include <fstream>
#include <random>
#include <cmath>
using namespace std;
double randnum (double aa, double bb)  //defining a function to create random numbers
{
  static std::default_random_engine generator;
  std::uniform_real_distribution<double> distribution (aa,bb);
  return distribution(generator);
}

int main() {
    double dr;
    double a = 1.0;               //minimum value for generated random number with power-law distribution
    int N = 100000;                  //number of time steps
    double u;                    //uniform random number used to create random number with desired distribution
    int Particles = 1000;       //number of particles
    int H = 500;                    //potential parameter
    int h = 200;                    //potential parameter
    double C = 0.004;                  //potential parameter
    double A = 0.002;              //Potential Parameter
    double Delta = 1/7;            //Assymetric Parameter
    double b = 25.0;               //maximum value for generated random number with power-law distribution
    double  phi;   //first angle
    std::vector<double>  x(Particles);
    std::vector<double>  pre_x(Particles);  //x_copmpnent of i'th particle position in previous step
    std::vector<double>  pre_y(Particles);
    std::vector<double>  J(N);
    std::vector<double>  y(Particles);
    for(int i = 0; i < Particles; i++)  //Initializing initial angle
        {
        pre_x[i] = 0;
        pre_y[i] = 200;
        }

    ofstream myfile;
    myfile.open ("J.txt");
    if(alph < 1.01 && alph > 0.99)
        B = 1  //renormalazation constant for distribution
        for(int i = 0; i < N; i++)  //time steps
            {
            J[i] = 0;
            for (int j = 0; j < Particles; j++)  //Particles
                {
                 u = randnum(0,1);
                 dr = pow( pow( a, 1-alph ) + u * (1-alph)/B, 1/(1-alph));
                 phi = randnum(0,M_PIl);
                 x[j] = pre_x[j] + cos(phi) * dr;
                 y[j] = pre_y[j] + sin(phi) * dr;
                 while( (sin(A * x[j]) + Delta * sin(C * x[j])/2) * h + H < y[j] || y[j] < 0)
                     {
                     u = randnum(0,1);
                     dr = pow( pow( a, 1-alph ) + u * (1-alph)/B, 1/(1-alph));
                     phi = randnum(0,M_PIl);
                     x[j] = pre_x[j] + cos(phi) * dr;
                     y[j] = pre_y[j] + sin(phi) * dr;
                     }
                 pre_x[j] = x[j];
                 pre_y[j] = y[j];
                 J[i] = J[i] + cos(phi);
                 }
                     myfile<<J[i]/Particles<<endl; //Outputs array to txtFile
            }
            myfile.close();
    }

Concerns:

code seems slow
general code quality

It is slow or in other words, I think is possible to make it faster. I need both "N" and "Particles" to be large values. With this condition, how can I make the code to run faster? Is there any change which could improve the run-time? Any other feedback is welcome as well.

Here are some things that may help you improve your code.

Don't abuse using namespace std

Putting using namespace std at the top of every program is a bad habit that you'd do well to avoid.  Once you remove that statement, you will have to add in the namespace where appropriate.  For example, fstream would be changed to std::fstream, endl to std:endl, etc.

Break up the code into smaller functions

In the current code, almost everything is done in main(). Rather than having everything in one long function, it would be easier to read and maintain if each discrete step were its own function.

Eliminate "magic numbers"

This code is littered with "magic numbers," that is, unnamed constants such as 0.004, 200, 25.0, etc.  Generally it's better to avoid that and give such constants meaningful names.  That way, if anything ever needs to be changed, you won't have to go hunting through the code for all instances of "200" and then trying to determine if this particular 200 is one that needs to be changed or just some other constant that happens to have the same value.

Consider improving names

I think randnum is not a bad function name, but most of the variable names are single letter names that are not at all descriptive of what they're intended to represent.  More descriptive variable names would help readers of the code understand what it's doing.

Write results to the std::cout

Rather than have a single hard-coded file name, write the data to std::cout.  If the user then wants to save a copy, doing so is then a simple matter of command line output redirection.

Use <cmath> rather than <math.h>

Use the new style <cmath> rather than the C-style <math.h> for two reasons.  First, it is more idiomatic modern C++, but also because it uses namespaces.  Consistent with that, you shoud also use std::cos rather than just cos.  It's the same for log, pow and sin.

Don't use std::endl when '\n' will do

Using std::endl emits a \n and flushes the stream.  Unless you really need the stream flushed, you can improve the performance of the code by simply emitting '\n' instead of using the potentially more computationally costly std::endl.

Use only necessary #includes

The #include <functional> line is not necessary and can be safely removed.

Use constexpr where practical

In main, almost all of the variables are actually used as constants, so it would make sense to at least declare them as const and preferably constexpr if you're using a C++11 compliant compiler (and if you're not, you really ought to update your compiler).

Use objects

The purpose of the code is to simulate objects called particles.  It's written in C++.  Why doesn't the code use C++ objects named Particle?  Here's a Particle class we can use:

class Particle {
public:
    Particle() : x_{0}, y_{200}, phi_{0} {}
    Particle &operator++();
    double phi() const { return phi_; }
public:
    static double get_dr();
    void next(const Particle &other);
    bool bad() const;
    double x_;
    double y_;
    double phi_;
};

Now main can look like this:

int main() {
    constexpr int N{10000};
    std::vector<double> J(N, 0);
    constexpr int numParticles{1000};
    std::vector<Particle> particles(numParticles);

    for (auto &step : J) {
        for (auto &p : particles) {
            ++p;
            step += std::cos(p.phi());
        }
        std::cout << step / numParticles << '\n';
    }
}

Isn't that nice?

Simplify the mathematics

Let's look at how the class functions above are implemented.  First is the operator++:

Particle &Particle::operator++() {
    Particle testp;
    for (testp.next(*this); testp.bad(); testp.next(*this))
    { /* do nothing */ }
    return *this = testp;
}

This has the effect of the inner loop in the original code.  The next function is this:

void Particle::next(const Particle &other) {
    double dr = get_dr();
    phi_ = randnum(0, M_PIl);
    x_ = static_cast<int>(other.x_) + std::cos(phi_) * dr;
    y_ = static_cast<int>(other.y_) + std::sin(phi_) * dr;
}

The get_dr function is this:

double Particle::get_dr() {
    constexpr double a = 1.0;
    constexpr double b = 25.0;
    constexpr int alph = 0;
    constexpr double B = (alph < 1.01 && alph > 0.99) ?
        1 / (std::log(b) * std::pow(b, 1 - alph) - std::log(a) * std::pow(a, 1 - alph))
       : (1 - alph) / (std::pow(b, (1 - alph)) - std::pow(a, (1 - alph)));
    return std::pow(std::pow(a, 1 - alph) + randnum(0, 1) * (1 - alph) / B, 1 / (1 - alph));
}

Finally the simple version of bad() is this:

bool Particle::bad() const {
    constexpr int H = 500;
    constexpr int h = 200;
    constexpr double C = 0.004;
    constexpr double A = 0.002;
    constexpr double Delta = 1 / 7;
    return
        (std::sin(A * x_) + Delta * std::sin(C * x_) / 2) * h + H < y_;
}

However, the maximum value of sin is 1, so the upper bound of the left hand side of that inequality can be expressed as a constant.  Since comparing against a constant is usually much faster than invoking std::sin we can use this knowledge as follows:

bool Particle::bad() const {
    constexpr int H = 500;
    constexpr int h = 200;
    constexpr double C = 0.004;
    constexpr double A = 0.002;
    constexpr double Delta = 1 / 7;
    constexpr double maxval = (1 + Delta * 1 / 2) * h + H;
    return maxval < y_ ||
        (std::sin(A * x_) + Delta * std::sin(C * x_) / 2) * h + H < y_;
}

The short circuit expression evaluation means we only invoke the computationally expensive sin function is the comparison against maxval fails.  Testing on my machine shows that this makes the program about 20% faster.  You could squeeze out still more time by noting that x need only be calculated if the first test fails.  Generally, you want to arrange things so that simpler tests are done first.

It may be advantageous to look at using a different coordinate scheme (polar vs. rectangular for instance) if the resulting equations simplify.  I suspect they might.


Testing uptime of my personal server I made some code to test my personal servers uptime. It logs in to my router page and sees if it returns status 200. A cron job runs it every 5 minutes. Please let me know if I can make the code more efficient.

<?php
//Date
date_default_timezone_set('America/Los_Angeles');
$date = date('m/d h:i:s a', time());

//Output
$myFile = dirname(__FILE__) . "/output.txt";
$fh = fopen($myFile, 'a+') or die("can't open file");

//Count #
$count_file = dirname(__FILE__) . "/OK.txt";
$fil = fopen($count_file, r);
$dat = fread($fil, filesize($count_file));
fclose($fil);
$fil = fopen($count_file, w);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "MYURL");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USER:PASSWORD");
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_UNRESTRICTED_AUTH, 1);
curl_exec($ch);
$retcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);

if($retcode==200){
    //fwrite($fh, "$date\tOK\n");
    $OK = $dat + 1;
    fwrite($fil, $OK);
    echo "$date\tOK\n";
    fclose($fh);
} else {
    fwrite($fh, "$date\t\tDOWN\n");
    echo "$date\tDOWN\n";
    fclose($fh);
}

//Count OK/DOWN
$read = file_get_contents($myFile);
$DOWN = preg_match_all("/DOWN/", $read, $matches) - 1;
$UP = 100*bcdiv($OK,$OK+$DOWN,4);

//Remove first line
$read = file($myFile);
array_shift($read);
$server_up = "Server Uptime: " . $UP . "%\nOK: " . $OK . ", DOWN: " . $DOWN . "\n";
$read[0] = $server_up;
file_put_contents($myFile, $read);

?>

One thing I wanted to do was to somehow assign $OK to a variable and retrieve it the next time the page is loaded but I do not think PHP can do this so I saved it to an external file (OK.txt) and I read the value every time and increment it every time my site returns 200.

Here is a sample output.txt file:

Server Uptime: 92.5%
OK: 37, DOWN: 3
09/30 06:48:28 pm DOWN
09/30 06:48:28 pm DOWN
09/30 06:48:28 pm DOWN

Naming

You should choose longer variable names. For example, with names as $fil and $fh it's hard to remember which one was which. You could name them $output and $count.

But all other names should be expressive as well. $dat might be $uptime_count, and $myFile -> $output_file_name, $fh -> $output_file, $count_file -> $count_file_name (it't just a string, not a file), $ch -> $curl, etc.

For some names it's obviously more important than for others, but good naming is always an improvement.

Variable Scoping

$fh is only used once, far away from where it was defined, inside the else statement. I would move the opening code here, so you don't have to think about $fh outside the else.

It's closed in the if as well, but that is unnecessary. Don't open up the file if you don't plan on writing to it.

Functions

The code isn't all that long, but I would still extract some functionality to its own function. Especially the counting of previous up and down times, and maybe the connecting to the server.