OAuth2 Authorization Using Python

Overview

As promised I am going to talk about getting the authorization from OAuth2. OAuth2 is a popular authorization protocol used by website like twitter, and other social websites in their APIs. Basic OAuth2 authorization involves two stages: first obtain authorization code, second obtain access_token.

Python Demo

Taking social website Douban as an example

import urllib
import requests
SCOPE = 'douban_basic_common,shuo_basic_r,shuo_basic_w' # getting more access by adding more parameters
APP_KEY = YOUR_APP_KEY # sometimes called API_KEY, obtain upon registration for API
APP_SECRET = YOUR_APP_SECRET # sometimes called API_SECRET, obtain upon registration for API
REDIRECT_URI = YOUR_CALLBACK_URI # this http address would be the address to receive authorization code, and access_token returned from website
DOUBAN_RESPONSE_TYPE = 'code'
DOUBAN_AUTHORIZATION_URI = "https://www.douban.com/service/auth2/"
DOUBAN_ACCESS_TOKEN_URI = "https://www.douban.com/service/auth2/"
v = {'client_id':APP_KEY, 'redirect_uri':REDIRECT_URI,'response_type':'code','scope':SCOPE}
params = urllib.urlencode(v)
url = DOUBAN_AUTHORIZATION_URI+params
# now, go to your browser, go to your generated url, click on accept authorization, then you would be redirect to REDIRECT_URI+code=YOUR_AUTHORIZATION_CODE, the YOUR_AUTHORIZATION_CODE would be appended at the end of that address
code = YOUR_AUTHORIZATION_CODE
# up to this point, you finished first step, obtain authorization code
v2 = {'client_id':APP_KEY, 'client_secret':APP_SECRET','redirect_uri':REDIRECT_URI, 'grant_type':'authorization_code','code':code}
url2 = DOUBAN_ACCESS_TOKEN_URI
r = requests.post(url2,data = v2)
r.status_code # should be 200 if everything is right
r_json = r.json()
# now you could see response in json format, with access_token in it

Just remember that first stage uses HTTP method GET, second stage uses HTTP method POST (that means you have to pass your encoded v2 to data, instead of params!!!)

Now you should have your access_token, go nuts!

A Little Note on Douban API

Overview

Douban is a popular Chinese social cultural site. It integrates Book reading, movie criticism, city events, and miniblog together.
From my past couple day’s experience, though most Chinese social websites open their API, but they all have different ‘tricks’ to successfully send your http post and get commands. I had been confused on Douban’s photo uploading format for a while, and now I was hoping to share the points you should be careful when uploading your photo using urllib and urllib2 libraries from python.

Encode your photo

The format to encode the photo to pass to Douban could be very restrict. The following format works:

------------0x14026e4be72\r\n
Content-Disposition: form-data; name="text"\r\n\r\nDraw\r\n------------0x14026e4be72\r\n
Content-Disposition: form-data; name="image"; filename="hidden"\r\n
Content-Length: 2019551\r\n
Content-Type: image/jpeg\r\n
Content-Transfer-Encoding: binary\n\r\n
YOUR_IMAGE BYTES
\r\n------------0x140297140d1\r\nContent-Disposition: form-data; name="source"\r\n\r\nYOUR_APP_KEY\r\n------------0x140297140d1--\r\n

Things you should really keep in mind is that Douban needs a clear declaration of Content-Type, you need to restrictedly follow the format. Use mimetypes.types_map would return you a dictionary of types they have, and you could pass your uploading file’s extension as a key into mimetypes.types_map.get(YOUR_FILE_EXTENTION), and it should return you the correct answer for Content-Type.
Also, 0x14026e4be72 is just the boundary variable you could generate randomly.

Following is the python code:
test_douban class

import gzip, time, urllib, urllib2, mimetypes

def _encode_multipart(**kw):
    ' build a multipart/form-data body with randomly generated boundary '
    boundary = '----------%s' % hex(int(time.time() * 1000))
    data = []
    for k, v in kw.iteritems():
        data.append('--%s' % boundary)
        if hasattr(v, 'read'):
            # file-like object:
            filename = getattr(v, 'name', '')
            content = v.read()
            data.append('Content-Disposition: form-data; name="%s"; filename="hidden"' % k)
            data.append('Content-Length: %d' % len(content))
            data.append('Content-Type: %s' % _guess_content_type(filename))
            data.append('Content-Transfer-Encoding: binary\n')
            data.append(content)
        else:
            data.append('Content-Disposition: form-data; name="%s"\r\n' % k)
            data.append(v.encode('utf-8') if isinstance(v, unicode) else v)
    data.append('--%s--\r\n' % boundary)
    return '\r\n'.join(data), boundary

def _guess_content_type(url):
    n = url.rfind('.')
    if n==(-1):
        return 'application/octet-stream'
    ext = url[n:].lower()
    # mimetyeps.types_map keys are stored in lower case, ex: '.jpeg'
    return mimetypes.types_map.get(ext, 'application/octet-stream')

This version of code is heavily depends on the existed code from internet. I just amended little bit on the encoding part to make the format be accepted by Douban.

call test_douban class

from test_douban import _encode_multipart
import urllib2
import urllib
url = 'https://api.douban.com/shuo/v2/statuses/'
access_token = YOUR_ACCESS_TOKEN
fh = open(PATH_TO_YOUR_IMAGE_FILE,'rb')
status = THINGS_YOU_WANT_TO_SAY
params = {'text':status, 'image':fh, 'source':YOUR_APP_KEY}
coded_params, boundary = _encode_multipart(**params)
# after previews call, you have the encoded photo, and a random generated boundary
req = urllib2.Request(url, data=coded_params)
req.add_header('Accept-Encoding', 'gzip')
req.add_header('Authorization', 'Bearer %s' % access_token)
req.add_header('Content-Type', 'multipart/form-data; boundary=%s' % boundary)
resp = urllib2.urlopen(req)

Now you should see the photo you just post on Douban’s miniblog. Cheers !

A Little Learning on API

Overview

API stands for Application Programming Interface. In the information age, there are huge amount of data generating everyday, most of them are being ignored and not substantially recognized for their values. A web API enables a third party program to access online data servers like NPR (national public radio) and sunlight foundation to gain news and political materials. A good idea of reorganizing, analysis those free open information could bring up potential economic value from what seems to be the chaotic data. In this post, I am discussing some basic programming to get access to NPR’s API, in the future, I may also post some more complex usage, like access your twitter API which requires oauth2 authentication.

Python code

Python offers wonderful library support for http usage
Python has several version of urllib library: urllib, urllib2, and requests (which is the third generation of urllib), I highly recommend the usage of requests for its simplicity and easy to understand.

import requests
import pprint
query_params = { 'apikey': 'YOUR API KEY',
		'phrase': 'holiday season',
		'start_date':'2012-12-12',
		'end_date':'2012-12-14',
        'state':'CA'}

endpoint = 'http://capitolwords.org/api/text.json'
response = requests.get( endpoint, params=query_params)
data = response.json()
pprint.pprint(data)
print data["num_found"]

This simple example is excerpt from the wonderful online programming tutorial website, Codecademy ,they offer various entry level programming practice for different programming language, and they also have some entry level API practices.
A little explanation to the code:
you get information you wanted by, basically, say piece together a unique URL, then use HTTP method either ‘post’ or ‘get’ (basic HTTP methods are ‘get’, ‘post’, ‘put’, and ‘delete’) you could receive the correspond information you requested, the format of the data package usually are JSON. Now, the unique URL we made, through the combination of ‘endpoint’ and ‘query_params’ is http://capitolwords.org/api/text.json?apikey=YOU_API_KEY&phrase=holiday+season&start_date=2012-12-12&end_date=2012-12-14&state=CA, this is basically telling the server that I am looking for any record, that a California senator spoke at congress between Dec 12, 2012 and Dec 14, 2012, and mentioned the phrase ‘holiday season’, and the server should return the record in JSON format.

a little note that, see the line where you call requests, you could use different HTTP methods just by changing the .get to .post following the requests command.

A big data era is ahead of us, now write your API programs, inspire yourself with more ideas of how data could be reorganized to serve our life better :)

Karaoke Game on Android

welcome page

gaming page

end of the game page

Overview

This application is an extension to the original spectrogram application. When people are singing to the microphone, the application calculating the autocorrelation of the sound recorded, it finds the pitch in his voice and compare it to the reference. The reference would be a recording of the piano play of the same song (piano would generate accurate pitch, thus this file, which recored and processed with MATLAB, could be serve as reference). Every matching between human voice and the reference pitch would gain one point, at the end of the song, a total score would be displayed.
Some key concept may be useful during the signal processing part are: autocorrelation , voice detection

MATLAB simulation

MATLAB main function call

numdatablc = size(a4)/256; % number of data blocks, each block has 256 points, store in variable a4
pdata1 = zeros(1,numdatablc); % initialize return array
for i=0:numdatablc-1
    temp = a4(i*256+1:(i+1)*256);
    if voidet(temp) % do voice detection
        pdata1(i+1) = proc(temp,f); % do autocorrelation
    else
        pdata1(i+1) = 0;
    end
end

MATLAB function for autocorrelation and pitch finding

function [ y ] = proc( x,f )
% f: sampling frequency
% filter
% f_pi/pi = f/4Khz
% f_pi = 2pi/range
% range = 8KHz/f = indx_max - indx_sec_max
% human vocal range: 75 ~ 500 Hz
% range: 8000/500 ~ 8000/75 => floor 16 ~ 107
lowr = floor(f/500); 
upr = ceil(f/75);
x_cor = xcorr(x);
x_filt = x_cor;
[xmax, po] = max(x_filt);
x_filt(1:lowr+po) = 0;
x_filt(po+upr+1:end) = 0;
[xmax1,pmax] = max(x_filt);
y = f/(pmax-po);
end

MATLAB function for voice detection

function [ y ] = voidet(x)
x2 = x.^2;
xs = sum(x2);
if xs > 50000000 %2.56
    y = 1;
else
    y = 0;
end
end

showing the calculated pitch from piano recording

alphabet song recording processed (show pitch vs. data-time), the middle simulation has been manually corrected to be more smooth to serve as reference file

Android app development

During the song playing, lyrics would scrolling though the screen, real-time calculated pitch and score displayed on the screen, and real-time spectrogram would be scrolling through the right part of the screen with the pitch (the fundamental frequency of a human voice, usually the lowest frequency among all the harmonics) mapped out on the screen in red.

This is really a funny project that combines signal processing theory with practical gaming experience.

This is the free choice of the final project of the Embedded Digital Signal Processing (DSP) lab, the source code of the project could be found here

Wildfire Monitor System: a project proposal

Every year we watched from the news about wildfire in western part of the US. It costed lots of money and human resource to fight those vicious natural disaster. After watching the news that 19 firefighters had died during a recent wildfire in Arizona, I started to think about how we could approach this problem from a technology angle.

Following graph shows the implementation of my solution to wildfire prevention and monitoring:

Design Overview

My design includes two major modules: the ground computer hub and the UAV monitor unit.
The UAV unit equips with:

• heat sensor
• GPS chip
• microprocessor
• RF transmitter
• High resolution camera

Computer hub features:

• RF receiver
• Human monitoring in front of computer
• Google Map API to help map out the ground of wildfire

Implementation of the project

The project would be implemented to each local forestry administration. Each location would have one ground unit, and one UAV in charge of sweeping a certain area.
At the beginning stage, UAV would proceed a preliminary sweep to map-out heat-spots from human habitation, and non-threaten source, the data from the first sweep would be processed with Google Map API to make a reference map.
Future routinely sweep data would be compared to this reference map on the fly. If a new heat-spot occurs, the UAV would take a picture of the scene and send the picture together with the GPS coordinates back the base station.
In base station, human worker would investigate the picture, assess the situation and make further decision on either to send out firefighter-jets to put out the fire.

Some interesting challenges may occur

• how to make more fuel-efficiency UAVs (solar panel, battery)
• design low energy consume GPS circuit & microprocessor
• most market existed GPS module output standard NMEA string, convert(A/D) and decipher this information into GPS coordinates
• design reliable RF communication between UAV and base
• implement Google Map API

Spectrogram on Android

A spectrogram could display several qualities of sound in a graphical way. It could map out the frequency of the sound component, color-code its intensity at each frequency.

As part of my Embedded DSP lab, we built this spectrogram program on Android platform. It records the sound, use Android NDK package to embed C code into Android program to process the sound with Fourier Transform and outputs its spectrogram.

pic 1

pic 1 shows the spectrogram of human voice, with different harmonics and intensity at each harmonic

The program’s sampling frequency is set at 8000Hz, thus it could detect highest frequency of 4000Hz without aliasing.

This is part of the Embedded Digital Signal Processing (DSP) lab, the code of this program could be found here

Image Processing on Android

This image processing app features histogram and RGB coloring.

The application’s main frame is developed on Android Eclipse platform (Android SDK), I also used a NDK package to embed C code in JAVA to do the image processing part.The program has a OpenCV dependency, as the program calls some computer vision class from the OpenCV library.The successful install and run the program on Android device, a OpenCV Manager from Google Play is also needed.

pic 1

This screenshot (pic 1) shows the app with the RGB color option chosen

pic 2

This screenshot (pic 2) shows the app with the histogram option chosen, as left half of the image has a clearer detail of the object in the far distance

This project is part of the Embedded Digital Signal Processing (DSP) class, the code could be found here .

Hello World

Hello World

(photo courtesy of Jesse van Dijk )

Shade

photo courtesy: Mary Altaffer/Associated Press

When it comes to travel with your family in the park on a sunny Sunday, people always come to appreciate that shaded area under the tree, or beside the building.

Art Duo Christo and Jeanne-Claude who are famous for their international series of large-sale outdoor installations. They used to took years to gain their permission to put their installations in central park.I admire their artistic spirit and appreciate their wonderful effort to bring inspirations to urban life.

They first proposed the project in 1979, but it was not approved until 2005.

The Gates, project for central park, New York City, 2005
Also on wikipedia and New York Times

Technology should also create a better living environment and inspire creativity.