Beautiful Python: Some Cool Language Constructs and Tricks for Beginners - Part 1


0. Pretty printing of a dictionary
Suppose you have a nested dictionary and you want a human readable view of it, you could use json module to accomplish this. json.dumps() takes an extra parameter 'indent' that formats the elements legibly.

>>> import json

>>> a = { 'a': {'b': {'c':'d', 'e':'f'}}}

>>> print json.dumps(a, indent=2)
{
  "a": {
    "b": {
      "c": "d",
      "e": "f"
    }
  }
}

You can also use the pprint python module for pretty print of python data structures.

1. Reverse an iterable in python

>>> a = [1, 2, 4]

>>> a[::-1]
[4, 2, 1]

>>> a
[1, 2, 4]

>>> b = (2, 3, 4)

>>> b[::-1]
(4, 3, 2)

>>> b
(2, 3, 4)

>>> c = "This is a string"

>>> c[::-1]
'gnirts a si sihT'

>>> c
'This is a string'
This method always returns a new instance of the iterable instead of an in-place reverse.

2. Swapping the values of two variables in python
>>> a = 1

>>> b=2

>>> a,b = b,a

>>> a
2

>>> b
1

How does this work?
Python separates the right-hand side expression from the left-hand side assignment. First the right-hand side is evaluated, and the result is stored on the stack, and then the left-hand side names are assigned using opcodes that take values from the stack again.
For tuple assignments with 2 or 3 items, Python just uses the stack directly:
>>> import dis
>>> def foo(a, b):
...     a, b = b, a
... 
>>> dis.dis(foo)
  2           0 LOAD_FAST                1 (b)
              3 LOAD_FAST                0 (a)
              6 ROT_TWO             
              7 STORE_FAST               0 (a)
             10 STORE_FAST               1 (b)
             13 LOAD_CONST               0 (None)
             16 RETURN_VALUE        
After the two LOAD_FAST opcodes (which push a value from a variable onto the stack), the top of stack holds [a, b]. The ROT_TWO opcode swaps the top two positions on the stack so the stack now has [b, a] at the top. The two STORE_FAST opcodes then takes those two values and store them in the names on the left-hand side of the assignment. The first STORE_FAST pops a value of the top of the stack and puts it into a, the next pops again, storing the value in b. The rotation is needed because Python guarantees that assignments in a target list on the left-hand side are done from left to right.
For rest of the answer refer: http://stackoverflow.com/questions/21047524/how-does-swapping-of-members-in-the-python-tuples-a-b-b-a-work-internally

3. Enumerate
When you loop through a sequence or an iterable, you can get the index and its corresponding value at the same time by wrapping the sequence in enumerate.

>>> for index, value in enumerate(['foo', 'bar', 'zoo']):
...     print index, value
...

0 foo
1 bar
2 zoo

4. Splitting a string into a list of words and join them back

To split a string by whitespace
>>> a = "This is a string"

>>> a.split()

['This', 'is', 'a', 'string']

To split a string by a character
>>> a = "This is a string"

>>> a.split('s')

['Thi', ' i', ' a ', 'tring']

To join a list of words by space
>>> b
['This', 'is', 'a', 'string']

>>> " ".join(b)
'This is a string'

To join a list of words by a character, comma for example

>>> b
['This', 'is', 'a', 'string']

>>> ",".join(b)
'This,is,a,string'


5. List Comprehensions
Suppose you have a list of elements and you need to do some operation on each of the element. For example, you have a list L consisting of words each of length greater than 5 and you have to create a new list consisting of first three letters of each word in L.
The common way to write code for this would be:

>>> L = ["Python", "makes", "people", "love her"]

>>> new_L = []

>>> for word in L:
...     new_L.append(word[0:3])
...

>>> new_L
['Pyt', 'mak', 'peo', 'lov']


This is where List comprehensions come to the rescue

>>> L = ["Python", "makes", "people", "love her"]

>>> new_L = [word[0:3] for word in L]

>>> new_L
['Pyt', 'mak', 'peo', 'lov']

This effectively reduced the number of lines from 3 in earlier approach to 1 using List comprehensions. Also, generally, List comprehensions are considered to be faster and efficient than creating an empty list and appending an element to that list one by one.

Now suppose you need only the words from L which have length>5.
>>> L = ["Python", "makes", "people", "love her"]

>>> new_L = [word for word in L if len(word)>5]

>>> new_L
['Python', 'people', 'love her']

Further reads on List Comprehensions:
[0] http://docs.python.org/2/tutorial/datastructures.html#list-comprehensions
[1] http://blog.cdleary.com/2010/04/efficiency-of-list-comprehensions/
[2] http://python-history.blogspot.in/2010/06/from-list-comprehensions-to-generator.html






Crawling a Website that loads content using Javascript with Selenium Webdriver in Python

Selenium is a browser automation tool that is used primarily for testing web applications. You can simulate real user actions and interactions with your web applications. Selenium supports all the major browser platforms  and operating systems. There are bindings for all the popular programming languages. The power of Selenium is not just restricted to testing your web apps, one other use can be of crawling or scraping websites, in particular, the ones which don't provide an API and load content lazily using Javascript.

Today, we will be crawling an online merchant website www.jabong.com with Selenium using its python bindings. Jabong loads more products as you scroll down a web page. We will use Selenium to simulate this user action of scrolling down a web page and then retrieving all the product titles and the corresponding links to the product detail pages.

Reference:

Platform: Ubuntu 12.04

Tools Required

sudo pip install selenium

sudo pip install xvfb

sudo pip install pyvirtualdisplay

We use pyvirtualdisplay which is a wrapper around xvfb and enables you to run Firefix headlessly.

Page to Crawl:




A quick "Inspect Element" on a shoe above shows that each of the product is wrapped by a "div" element with class "hover-box" and the title and links are embedded in an "a" element within those "div" elements.



Code:



Output
 javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Liteforce-Ii-Mid-White-Sneakers-503958.html','6')
Nike Ballista Iv Msl Grey Running Shoes http://www.jabong.com/Nike-Ballista-Iv-Msl-Grey-Running-Shoes-503907.html
U.S. Polo Assn. Navy Blue Sneakers http://www.jabong.com/Us_Polo_Assn-Navy-Blue-Sneakers-518870.html
Nike Dewired Navy Blue Sneakers http://www.jabong.com/Nike-Dewired-Navy-Blue-Sneakers-503897.html
 javascript:sendSizeFormWithSize('http://www.jabong.com/Us_Polo_Assn-Black-Dress-Shoes-518868.html','6')
 javascript:sendSizeFormWithSize('http://www.jabong.com/Us_Polo_Assn-Destiny-Navy-Blue-Sneakers-518878.html','6')
United Colors of Benetton Black Boat Shoes http://www.jabong.com/United-Colors-of-Benetton-Black-Boat-Shoes-512546.html
Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506097.html
 javascript:sendSizeFormWithSize('http://www.jabong.com/United-Colors-of-Benetton-Blue-Sneakers-512548.html','6')
U.S. Polo Assn. Delta Beige Sneakers http://www.jabong.com/Us_Polo_Assn-Delta-Beige-Sneakers-518873.html
Asics Kayano 20 Black Running Shoes http://www.jabong.com/Asics-Kayano-20-Black-Running-Shoes-507673.html
Nike Air Max 2014 Blue Running Shoes http://www.jabong.com/Nike-Air-Max-2014-Blue-Running-Shoes-503964.html
Asics Excel 33 3 Navy Blue Running Shoes http://www.jabong.com/Asics-Excel-33-3-Navy-Blue-Running-Shoes-507674.html
Nike Air Relentless 3 Msl Blue Running Shoes http://www.jabong.com/Nike-Air-Relentless-3-Msl-Blue-Running-Shoes-503954.html
Asics Kayano 20 Navy Blue Running Shoes http://www.jabong.com/Asics-Kayano-20-Navy-Blue-Running-Shoes-507672.html
Nike Lunarinternationalist Grey Running Shoes http://www.jabong.com/Nike-Lunarinternationalist-Grey-Running-Shoes-503975.html
Nike Free 5.0+ Blue Running Shoes http://www.jabong.com/Nike-Free-50-Blue-Running-Shoes-503919.html
Nike Fs Lite Run Black Running Shoes http://www.jabong.com/Nike-Fs-Lite-Run-Black-Running-Shoes-503955.html
Nike Lunarinternationalist Blue Running Shoes http://www.jabong.com/Nike-Lunarinternationalist-Blue-Running-Shoes-503976.html
Nike Free Trainer 5.0 Grey Running Shoes http://www.jabong.com/Nike-Free-Trainer-50-Grey-Running-Shoes-503886.html
 javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Zoom-Terra-Kiger-Blue-Running-Shoes-503930.html','6')
Andrew Hill Brown Dress Shoes http://www.jabong.com/andrew-hill-Brown-Dress-Shoes-506110.html
Nike Lunar Forever 3 Msl Grey Running Shoes http://www.jabong.com/Nike-Lunar-Forever-3-Msl-Grey-Running-Shoes-503970.html
Asics Kayano 20 Red Running Shoes http://www.jabong.com/Asics-Kayano-20-Red-Running-Shoes-507671.html
Nike Free Trainer 5.0 Black Running Shoes http://www.jabong.com/Nike-Free-Trainer-50-Black-Running-Shoes-503915.html
Nike Chroma Thong Iii Green Slippers http://www.jabong.com/Nike-Chroma-Thong-Iii-Green-Slippers-503977.html
Nike Aquahype Blue Flip Flops http://www.jabong.com/Nike-Aquahype-Blue-Flip-Flops-503890.html
Z Collection Green Loafers http://www.jabong.com/z-collection-Green-Loafers-517513.html
Nike Flex 2013 Rn Black Running Shoes http://www.jabong.com/Nike-Flex-2013-Rn-Black-Running-Shoes-503916.html
U.S. Polo Assn. Brown Sneakers http://www.jabong.com/Us_Polo_Assn-Brown-Sneakers-518869.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506093.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506102.html
Nike Eliminate Ii Leather Grey Sneakers http://www.jabong.com/Nike-Eliminate-Ii-Leather-Grey-Sneakers-503909.html
Nike Fs Lite Trainer Blue Running Shoes http://www.jabong.com/Nike-Fs-Lite-Trainer-Blue-Running-Shoes-503952.html
Nike Suketo 2 Leather Red Sneakers http://www.jabong.com/Nike-Suketo-2-Leather-Red-Sneakers-503972.html
Nike Free Flyknit+ Red Running Shoes http://www.jabong.com/Nike-Free-Flyknit-Red-Running-Shoes-469765.html
 javascript:sendSizeFormWithSize('http://www.jabong.com/Nike-Zoom-Wildhorse-Blue-Running-Shoes-503931.html','6')
Nike Air Pegasus+ 30 Grey Running Shoes http://www.jabong.com/Nike-Air-Pegasus-30-Grey-Running-Shoes-503934.html
Nike Flex Supreme Tr 2 Grey Running Shoes http://www.jabong.com/Nike-Flex-Supreme-Tr-2-Grey-Running-Shoes-503942.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506088.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506087.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506091.html
Nike Zoom Structure+ 17 Black Running Shoes http://www.jabong.com/Nike-Zoom-Structure-17-Black-Running-Shoes-503949.html
Nike Flyknit Lunar2 Black Running Shoes http://www.jabong.com/Nike-Flyknit-Lunar2-Black-Running-Shoes-503959.html
Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506103.html
Andrew Hill Black Dress Shoes http://www.jabong.com/andrew-hill-Black-Dress-Shoes-506109.html
Phosphorus Tan Loafers http://www.jabong.com/phosphorus-Tan-Loafers-506095.html
Phosphorus Black Loafers http://www.jabong.com/phosphorus-Black-Loafers-506090.html
U.S. Polo Assn. Delta Navy Blue Sneakers http://www.jabong.com/Us_Polo_Assn-Delta-Navy-Blue-Sneakers-518872.html
Phosphorus Brown Loafers http://www.jabong.com/phosphorus-Brown-Loafers-506094.html
Nike Lunar Forever 3 Msl White Running Shoes http://www.jabong.com/Nike-Lunar-Forever-3-Msl-White-Running-Shoes-503971.html
Asics Kayano 20 White Running Shoes http://www.jabong.com/Asics-Kayano-20-White-Running-Shoes-507670.html

Resources for Learning Number Theory for Competitive Programming

Here are few resources for learning number theory for competitive programming.

Online resources:
1. Stanford Course: CS 97SI: Introduction to Competitive Programming Contests
2. Codechef: Basic Number Theory Every Programmer Should Know..
3. Topcoder: Primality Testing: Non Deterministic Algorithms
4. Topcoder: Prime Numbers, Factorization and Euler Function
5. MAXimal Russian Site. Contains the implementations of all major algorithms used in competitive programming contests

Books:
1. Elementary Number Theory by David Burton
    DescriptionElementary Number Theory, Seventh Edition, is written for the one-semester undergraduate number theory course taken by math majors, secondary education majors, and computer science students. This contemporary text provides a simple account of classical number theory, set against a historical background that shows the subject's evolution from antiquity to recent research. Written in David Burton’s engaging style, Elementary Number Theory reveals the attraction that has drawn leading mathematicians and amateurs alike to number theory over the course of history.

2. Number Theory for Computing
     DescriptionThis book provides a good introduction to the classical elementary number theory and the modern algorithmic number theory, and their applications in computing and information technology, including computer systems design, cryptography and network security. In this second edition proofs of many theorems have been provided, further additions and corrections were made

Top Sites to learn and practice programming

 Singpath
SingPath is the most FUN way to practice software languages! SingPath provides a platform to those that want to test their programming skills in a competitive and fun environment. There is a range of programming languages that you can test your skills with including Python, JavaScript, Ruby, Objective C and Java with many other languages to come. Specialize in one language or show your flexibility and skill by coding in multiple languages.
 SingPath

Udacity

Add caption


Coderbyte



LearnStreet



Khan Academy



CodeHS



treehouse



CodeSchool


How to take set difference of two sets in C++?

Set difference of two sets A and B is defined as the set C consisting of elements from A not present in B.

In C++, algorithm header in STL defines a function set_difference which in conjunction with inserter can be used to take set difference of two sorted range of elements.

Example:

#include<algorithm>

#include<set>

using namespace std;

..

set<int> A, B, C;

..

set_difference(A.begin(), A.end(), B.begin(), B.end(), inserter(C, C.end()))




So, this basically takes the set difference of the whole of A and B and copies those elements to C.

set_difference can even work on vectors and arrays. Just make sure that the vectors and arrays are sorted before you take the set difference.

Complexity: O(n)



How to convert a byte array into a string in C#?

In applications where you have binary data in a byte array for e.g. data received via a SocketStream, and you need to convert it to a string to interpret it, here is how you do it:
string result = System.Text.Encoding.UTF8.GetString(byteArray)