Python: 2015

Python: Extracting json and inserting into sqllite3

This is an interesting project that combining python, json and sqllite to extract, store and processing the data.

The first part are recreating the required tables. Second, part is to read and extract the roster1 json file and capturing the records and arranging them in columns (name, title and role). Nowthat we have the records for all 3 columns and inserting them into all dedicated tables.

Codes are presented in screenshots on purpose to deter from easy google searching for solution as this is a course assignment.

Here is an example of how does the role table looks like.

What I learned here are the embedded SQL in the executescript and the anchoring of the binding variables. With this simple algorithm and know-how, I can do a lot of more complicated applications.

Python: Raspberry Pi: Adding Button to control LED blinking

Second Project

This is an extension of the first project where letting the python to control the LED blinking. Instead of that, I am going to a physical button to control the LED blinking instead.

Requirements are, when button not pressed, the LED blinks and when the button pressed, the LED lid continuously.

This project is slightly trickier as more wires involved but not terribly bad. Just be cautious of where the wires go.

I am rewiring everything from the first project as I need to cram the LED and the button in the tiny breadboard.

My new pin assignments are the following
pin # 27: LED
pin # 4 : Button

Picture of the breadboard

Code:

https://youtu.be/UARP5Cuh4FY

Python: Raspberry Pi: First project: LED Blinking

First project

This is an easy project that using the Raspberry Pi GPIO 40 pin to control the LED blinking with Python GPIO library.

I would not be going through the process of putting the wires together on the breadboard. Picture tells it all. Just remember to ground it and have a resistor.

I am using pin 4 for the LED

Code:

import RPi.GPIO as GPIO

import time

GPIO.setmode(GPIO.BCM)

GPIO.setup(4, GPIO.OUT)

while True:

GPIO.output(24, True)

print "Led On"

time.sleep(3)

GPIO.output(4, False)

print "Led Off"

time.sleep(2)

Not finding oracle libraries

Just an indication that the python cx_Oracle did not find the libraries.

[oracle@localhost samples]$ python vpxadmin.py Traceback (most recent call last):

File "vpxadmin.py", line 1, in <module>

import cx_Oracle

ImportError: libclntsh.so.11.1: cannot open shared object file: No such file or directory [oracle@localhost samples]$ python vpxadmin.py Traceback (most recent call last):

File "vpxadmin.py", line 1, in <module>

import cx_Oracle

ImportError: libclntsh.so.11.1: cannot open shared object file: No such file or directory [oracle@localhost samples]$ . oraenv ORACLE_SID = [oracle] ? orcl The Oracle base for

ORACLE_HOME=/home/oracle/app/oracle/product/11.2.0/dbhome_1 is /home/oracle/app/oracle

Just do the following. Assuming orcl is your instance name which is also a default.

. oraenv

orcl

Install cx_Oracle

Download the gz file and extract it.
http://cx-oracle.sourceforge.net/

Switch to Oracle user as it will need Oracle libraries to perform the installation.
su - oracle

important: switch to the oracle environment you would like to install this. By doing it, . oraenv, it will locate the right Oracle Home and knowing where all the libraries reside. My instance is "orcl"

. oraenv

orcl

The installation part.

python setup.py build

pyhton setup.py install

During the cx_Oracle copying stage, it might fail to copy (cx_Oracle.so) specific libraries path to the python directories. Just perform a chmod 755 on the python package_site path with root if the python has been installed with root previously.

That's it.

There are some examples in the "samples" directory you can try it out. The DatabaseStartup.py and DatabaseShutdown.py are pretty cool.

How many countries do not have "a" in their names?

Saw the question posted online and decided to see if I can solve the question with Python programming.

30 minute later ....

source: http://www.countries-list.info/Download-List/download

Setup, copy the contents in the link above and and save it as country3.xml file in the same directory where the python script resides.

import urllib

import xml.etree.ElementTree as ET

data = urllib.urlopen('country3.xml').read()

tree = ET.fromstring(data)

lst = tree.findall('./option') #to get country counts and lists

convert_string__to_list = list()

cnt = 1

for item in lst:

if 'a' not in item.get('label').lower():

convert_string__to_list.append(item.get('label'))

print cnt, ")", item.get('label')

cnt = cnt + 1

print "Number of countries without 'a' in the name are ", cnt-1, "out of " , len(lst) , "countries."

Output should look like this ..

1 ) Belgium

2 ) Belize

3 ) Benin

4 ) Brunei

5 ) Burundi

6 ) Chile

7 ) Comoros

8 ) Cyprus

9 ) Czech Republic

10 ) Cote d%’Ivoire

11 ) Djibouti

12 ) Egypt

13 ) Fiji

14 ) French Southern Territories

15 ) Greece

16 ) Guernsey

17 ) Jersey

18 ) Lesotho

19 ) Liechtenstein

20 ) Luxembourg

21 ) Mexico

22 ) Montenegro

23 ) Morocco

24 ) Niger

25 ) Niue

26 ) Peru

27 ) Philippines

28 ) Puerto Rico

29 ) Réunion

30 ) Seychelles

31 ) Sweden

32 ) Timor-Leste

33 ) Togo

34 ) Turkey

35 ) United Kingdom

36 ) Yemen

Number of countries without 'a' in the name are 36 out of 265 countries.

Easier way to get BeautifulSoup working

Taking a Python class on web crawling topic. Professor showed that there is an easier way to get the BeautifulSoup working by simply copying the python file and put it in the same directory. No installation necessary. As it is called within the program, it will create its own compiled file on the fly.

http://www.pythonlearn.com/code/BeautifulSoup.py

Just like the following. Upon executing, "python Week4a", it will pick up the BeautifulSoup on the fly.

Enjoy and happy coding!

Problem with installing beautifulsoup with pip

Trying to install python beautifulsoup for the purpose of web scraping. Unfortunately, pip installed it successfully but module cannot be located when called.

C:\pip install beautifulsoup4
You are using pip version 6.0.6, however version 7.1.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting beautifulsoup4
Downloading beautifulsoup4-4.4.0-py2-none-any.whl (81kB)
100% |################################| 81kB 152kB/s ta 0:00:011
Installing collected packages: beautifulsoup4

Successfully installed beautifulsoup4-4.4.0

C:\pip install BeautifulSoup
You are using pip version 6.0.6, however version 7.1.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting BeautifulSoup
Downloading BeautifulSoup-3.2.1.tar.gz
Installing collected packages: BeautifulSoup
Running setup.py install for BeautifulSoup
Successfully installed BeautifulSoup-3.2.1

C:\>

Traceback (most recent call last):

File "C:\Scrap1.py", line 1, in <module>
from bs4 import BeautifulSoup
ImportError: No module named bs4
>>>

Download and install it manually.

http://www.crummy.com/software/BeautifulSoup/bs4/download/

C:\beautifulsoup4-4.4.0\beautifulsoup4-4.4.0>python setup.py install
C:\Python27\lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'use_2to3'
warnings.warn(msg)
running install
running bdist_egg
running egg_info
writing requirements to beautifulsoup4.egg-info\requires.txt
writing beautifulsoup4.egg-info\PKG-INFO
writing top-level names to beautifulsoup4.egg-info\top_level.txt
writing dependency_links to beautifulsoup4.egg-info\dependency_links.txt
reading manifest file 'beautifulsoup4.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'beautifulsoup4.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
creating build
creating build\lib
creating build\lib\bs4
copying bs4\dammit.py -> build\lib\bs4
copying bs4\diagnose.py -> build\lib\bs4
copying bs4\element.py -> build\lib\bs4
copying bs4\testing.py -> build\lib\bs4
copying bs4\__init__.py -> build\lib\bs4
creating build\lib\bs4\builder
copying bs4\builder\_html5lib.py -> build\lib\bs4\builder
copying bs4\builder\_htmlparser.py -> build\lib\bs4\builder
copying bs4\builder\_lxml.py -> build\lib\bs4\builder
copying bs4\builder\__init__.py -> build\lib\bs4\builder
creating build\lib\bs4\tests
copying bs4\tests\test_builder_registry.py -> build\lib\bs4\tests
copying bs4\tests\test_docs.py -> build\lib\bs4\tests
copying bs4\tests\test_html5lib.py -> build\lib\bs4\tests
copying bs4\tests\test_htmlparser.py -> build\lib\bs4\tests
copying bs4\tests\test_lxml.py -> build\lib\bs4\tests
copying bs4\tests\test_soup.py -> build\lib\bs4\tests
copying bs4\tests\test_tree.py -> build\lib\bs4\tests
copying bs4\tests\__init__.py -> build\lib\bs4\tests
creating build\bdist.win-amd64
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\bs4
creating build\bdist.win-amd64\egg\bs4\builder
copying build\lib\bs4\builder\_html5lib.py -> build\bdist.win-amd64\egg\bs4\builder
copying build\lib\bs4\builder\_htmlparser.py -> build\bdist.win-amd64\egg\bs4\builder
copying build\lib\bs4\builder\_lxml.py -> build\bdist.win-amd64\egg\bs4\builder
copying build\lib\bs4\builder\__init__.py -> build\bdist.win-amd64\egg\bs4\builder
copying build\lib\bs4\dammit.py -> build\bdist.win-amd64\egg\bs4
copying build\lib\bs4\diagnose.py -> build\bdist.win-amd64\egg\bs4
copying build\lib\bs4\element.py -> build\bdist.win-amd64\egg\bs4
copying build\lib\bs4\testing.py -> build\bdist.win-amd64\egg\bs4
creating build\bdist.win-amd64\egg\bs4\tests
copying build\lib\bs4\tests\test_builder_registry.py -> build\bdist.win-amd64\egg\bs4\tests
copying build\lib\bs4\tests\test_docs.py -> build\bdist.win-amd64\egg\bs4\tests
copying build\lib\bs4\tests\test_html5lib.py -> build\bdist.win-amd64\egg\bs4\tests
copying build\lib\bs4\tests\test_htmlparser.py -> build\bdist.win-amd64\egg\bs4\tests
copying build\lib\bs4\tests\test_lxml.py -> build\bdist.win-amd64\egg\bs4\tests
copying build\lib\bs4\tests\test_soup.py -> build\bdist.win-amd64\egg\bs4\tests
copying build\lib\bs4\tests\test_tree.py -> build\bdist.win-amd64\egg\bs4\tests
copying build\lib\bs4\tests\__init__.py -> build\bdist.win-amd64\egg\bs4\tests
copying build\lib\bs4\__init__.py -> build\bdist.win-amd64\egg\bs4
byte-compiling build\bdist.win-amd64\egg\bs4\builder\_html5lib.py to _html5lib.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\builder\_htmlparser.py to _htmlparser.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\builder\_lxml.py to _lxml.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\builder\__init__.py to __init__.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\dammit.py to dammit.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\diagnose.py to diagnose.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\element.py to element.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\testing.py to testing.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\tests\test_builder_registry.py to test_builder_registry.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\tests\test_docs.py to test_docs.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\tests\test_html5lib.py to test_html5lib.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\tests\test_htmlparser.py to test_htmlparser.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\tests\test_lxml.py to test_lxml.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\tests\test_soup.py to test_soup.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\tests\test_tree.py to test_tree.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\tests\__init__.py to __init__.pyc
byte-compiling build\bdist.win-amd64\egg\bs4\__init__.py to __init__.pyc
creating build\bdist.win-amd64\egg\EGG-INFO
copying beautifulsoup4.egg-info\PKG-INFO -> build\bdist.win-amd64\egg\EGG-INFO
copying beautifulsoup4.egg-info\SOURCES.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying beautifulsoup4.egg-info\dependency_links.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying beautifulsoup4.egg-info\requires.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying beautifulsoup4.egg-info\top_level.txt -> build\bdist.win-amd64\egg\EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist\beautifulsoup4-4.4.0-py2.7.egg' and adding 'build\bdist.win-amd64\egg' to it
removing 'build\bdist.win-amd64\egg' (and everything under it)
Processing beautifulsoup4-4.4.0-py2.7.egg
Copying beautifulsoup4-4.4.0-py2.7.egg to c:\python27\lib\site-packages
Adding beautifulsoup4 4.4.0 to easy-install.pth file

Installed c:\python27\lib\site-packages\beautifulsoup4-4.4.0-py2.7.egg
Processing dependencies for beautifulsoup4==4.4.0
Finished processing dependencies for beautifulsoup4==4.4.0

Tested and no more error.

C:\>python
Python 2.7 (r27:82525, Jul 4 2010, 07:43:08) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from bs4 import BeautifulSoup
>>> quit()

But lxml failling now when running my python web scrapper codes.

Traceback (most recent call last):
File "C:\Scrap1.py", line 31, in <module>
    categories = get_category_links(food_n_drink)
File "C:\Scrap1.py", line 12, in get_category_links
    soup = make_soup(section_url)
File "C:\Scrap1.py", line 9, in make_soup
    return BeautifulSoup(html, "lxml")
File "build\bdist.win-amd64\egg\bs4\__init__.py", line 156, in __init__
    % ",".join(features))
FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?
>>>

C:\>pip install --upgrade lxml
You are using pip version 6.0.6, however version 7.1.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting lxml
Downloading lxml-3.4.4-cp27-none-win_amd64.whl (3.3MB)
100% |################################| 3.3MB 660kB/s ta 0:00:01
Installing collected packages: lxml

Successfully installed lxml-3.4.4

C:\>pip install cssselect
You are using pip version 6.0.6, however version 7.1.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting cssselect
Downloading cssselect-0.9.1.tar.gz
Installing collected packages: cssselect
Running setup.py install for cssselect
Successfully installed cssselect-0.9.1

Reinstalling did not take care of it.

Download and install it manually fixed my issue.

https://pypi.python.org/pypi/lxml/3.2.5

Download and install https://pypi.python.org/packages/2.6/l/lxml/lxml-3.2.5.win-amd64-py2.6.exe#md5=90d59a70db1ab0bce2425d0de2aaa0da

Just some python practices

Nothing much in this blog. Just some an initiative for re-enhance my learning of Python.

First project

Some good python learning links

Online Compiler

Python Download

Canopy (download python either 2.7 or 3.x version first)