Web Service   and Open Data

(1)

Web Service   and Open Data

By Quentin Bramas - ProgRes 2016 quentin.bramas@lip6.fr

What is a Web Service ?

A Web Service is a method of communication between two electronic devices over the Web.

HTTP is the typical protocol used by WebService to communicate.

What is a Web Service ?

Request

Response

Device HTTP Server

What is a Web Service ?

Request

Response

Device HTTP Server

XML:

<id>5</id>

<note id=‘5’>

<to>Tove</to>

<from>Jani</from>

<heading>Reminder</heading>

<body>Don't forget the diner</body>

</note>

(2)

What is a Web Service ?

Request

Response

Device HTTP Server

SOAP:

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">

<soap:Header>

</soap:Header>

<soap:Body>

<m:GetStockPrice xmlns:m="http://www.example.org/stock/Surya">

<m:StockName>IBM</m:StockName>

</m:GetStockPrice>

</soap:Body>

</soap:Envelope>

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">

<soap:Header>

<requestId>xxxxxxxxxxxxxxxxxxxx</requestId>

</ResponseHeader>

</soap:Header>

<soap:Body>

<rval>

<name>RootAdUnit</name>

<status>ACTIVE</status>

What is a Web Service ?

Request

Response

Device HTTP Server

Url Encoded:

JSON:

order=date&limit=2

{

"data": [{

"id": 1001, "name": "Jim"

}, {

"id": 1002, "name": "Matt"

}]

}

REST Web Api

Is a web service using simpler REpresentational State Transfer (REST) based communication.

Request is just a HTTP Method over an URI. 

Response is typically JSON or XML.

Exemple:

GET : http://pokeapi.co/api/v1/pokemon/25

HTTP Method URI that represents a resource base URL of the API API version

REST API Call Example

{ 

"name": "Pikachu",  "attack": 55,  "abilities": [  { 

"name": "static", 

"resource_uri": "/api/v1/ability/9/" 

},  { 

"name": "lightningrod", 

}  ]  } GET /api/v1/pokemon/25/ HTTP/1.1

Host: pokeapi.co Connection: keep-alive Pragma: no-cache Cache-Control: no-cache

Accept: application/json,;q=0.9,*/

*;q=0.8

Accept-Encoding: gzip, deflate, sdch

HTTP Request Headers

HTTP Response Headers

HTTP/1.1 200 OK Server: nginx/1.1.19

Date: Fri, 08 Jan 2016 13:10:08 GMT Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Vary: Accept

X-Frame-Options: SAMEORIGIN

Cache-Control: s-maxage=360, max-age=360

HTTP Response Body

(3)

REST API Call Example

{ 

"name": "Pikachu",  "attack": 55,  "abilities": [  { 

"name": "static", 

},  { 

"name": "lightningrod", 

}  ]  } GET /api/v1/pokemon/25/ HTTP/1.1

Host: pokeapi.co Connection: keep-alive Pragma: no-cache Cache-Control: no-cache

Accept: application/json,;q=0.9,*/

*;q=0.8

Accept-Encoding: gzip, deflate, sdch

HTTP Request Headers

HTTP Response Headers

HTTP/1.1 200 OK Server: nginx/1.1.19

Date: Fri, 08 Jan 2016 13:10:08 GMT Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Vary: Accept

X-Frame-Options: SAMEORIGIN

Cache-Control: s-maxage=360, max-age=360

HTTP Response Body

Resources

Command based (ex: Flicker Api): 

GET:  

https://api.flickr.com/services/rest/?method=flickr.galleries.getList&user_id=XX

POST:  

https://api.flickr.com/services/rest/?method=flickr.galleries.addPhoto&gallery_id=XX 

 

Bad Api Rest design

Resources

• ex: Facebook Graph Api: 

GET: /{photo-id} to retrieve the info of a photo  

GET: /{photo-id}/likes to retrieve the people who like it  POST: /{photo-id} to update the photo 

DELETE : /{photo-id} to delete the photo 

 

URI/Resource based:

• ex: Google Calendar Api: 

GET: /calendars/{calendarId} to retrieve the info of a calendar  

PUT: /calendars/{calendarId} to update a calendar  DELETE : /calendars/{calendarId} to delete a calendar  POST: /calendars to create a calendar  GET: /calendars/{calendarId}/events/{eventId} 

 

Response

HTTP Response:

•

200: OK

•

3 _ _: Redirection

•

404: not found

(4 _ _ : something went wrong with what you try to access)

•

5 _ _ : Server Error

API Response:

•

Flickr: 

{ "stat": "fail", "code": 1, "message": "User not found" }

 

{ "galleries": { ... }, "stat": "ok" }

• Google Calendar: 

{ "error": {"code": 403, "message": "User Rate Limit Exceeded" } }  { "kind": "calendar#events","summary": ..., "description": ...

(4)

•

text/plain

•

text/html

•

text/xml or application/xml

•

application/json

•

image/png

•

...

Response

Content-Type:

JSON and XML   Parsing

use the json package:

>> obj = json.loads('{"attr1": "v1", "attr2": 42}') 

>> obj['attr1'] 

'v1' 

>> obj['attr2']  

42 >> obj = {'id':1, 'data':[1,2,3,4]} 

>> json.dumps(obj) # returns a string  '{'id':1, 'data':[1,2,3,4]}'

JSON Parsing XML Parsing

With xml.etree.ElementTree, xml.sax, or html.parser

import xml.etree.ElementTree as ET

tree = ET.parse('country_data.xml')

xml.etree.ElementTree load the whole file, you can

then naviguate in the tree structure.

(5)

XML Parsing

With xml.etree.ElementTree, xml.sax, or html.parser

from html.parser import HTMLParser class MyHTMLParser(HTMLParser):

def handle_starttag(self, tag, attrs):

print("start tag:", tag) def handle_endtag(self, tag):

print("end tag :", tag) parser = MyHTMLParser()

xml.sax and html.parser read the file once, and your functions are called when a element starts or ends.

XML Parsing

import xml.sax

class MyHandler(xml.sax.ContentHandler):

def init(self): 

self.inH1 = False  self.title = ''

def startElement(self, name, attrs):

if name == "h1":

self.inH1 = True

def endElement(self, name, attrs):

if name == "h1":

self.inH1 = False  def characters(self, content):

if self.inH1: 

self.title += content

parser = xml.sax.make_parser()  handler = MyHandler()

parser.setContentHandler(handler) parser.parse(open("test.xml","r"))  print(handler.title)

Create an API with Python

Create an API

• Django: Powerful web framework with a lot of modules. Great to build a complete website.

• Flask: Small Framework to build simple website.

• Bottle: Similar to Flask, but even simpler. Perfect to build an API

Available library/framework in python:

(6)

Create an API

The Bottle Framework (single file module, no dependencies)

• Routing: Requests to function-call mapping with support for clean and dynamic URLs.

• Templates: Fast and pythonic built-in template engine

• Utilities: Convenient access to form data, file uploads, cookies, headers and other HTTP-related metadata.

• Server: Built-in HTTP development server and

support for other WSGI capable HTTP server.

(WSGI is the Web Server Gateway Interface, which is a specification for web server in python)

Create an API

from bottle import route, run, template

@route('/hello/<name>') def index(name):

return 'Hello '+name

run(host='localhost', port=8080) Hello world example:

Create an API

@route('/get/<id:int>') def get(id):

Filters: :int, :float, :path, :re

/get/1

@route('/get/<filepath:path>') def get(filepath):

/get/images/sun.png

filepath

@route('/get/<idSlug:re:[0-9]+-.*>') def get(idSlug):

/get/51-bottle

Create an API

@route('/get/<id:int>')

HTTP Request Methods

accept GET method

@post('/images')

@route('/images', method='POST') or

Same thing with PUT and DELETE

(7)

Create an API

@route('/get/<id:int>'):

def get(id):

Generating the Content:

•

return no content (False, empty string, ...) return False

•

return a string return 'Hello' 

return json.dumps(dictObject)

•

return a file

return open('file.ext', 'rb') 

return bottle.static_file('file.ext') # Optimized by bottle

•

return an iterable

return ['line1\n', 'line2\n']

yield 'line1\n' yield 'line2\n'

Create an API

from bottle import route, response 

@route('/get/<id:int>'):

def get(id):

response.content_type = 'application/json; charset=utf-8'  return json.dumps({'id':id, 'data': ...}) 

When returning a string (or iterable of string), it is automatically encoded with respect to the given content-type, because only bytes are sent back to the client.

The Bottle Response Object:

Here, the server send a byte string encoded in utf-8, that represents an object formated in json.

Create an API

from bottle import route, abort 

@route('/get/<id:int>'):

def get(id):

abort(401, "Sorry, access denied.")

Sending Error:

@route('/getImage/<id:int>'):

def getImage(id):

redirect("/get/"+id)

Redirect:

Create an API

from bottle import route, response, request 

@route('/getAll'):

def getAll(): 

# catch params in the url: /getAll?order=name order = request.query['order'] 

# catch params sent by forms (with POST method) order = request.forms['order']  

# catch all params (sent by forms or in the url) order = request.params['order']  

# catch files sent by forms (with POST method)  image = request.files['image']

image.save('images/uploaded')

Request Data:

(8)

Test Your API

With the requests package:

>>> r = requests.get('http://pokeapi.co/api/v1/pokemon/25')

>>> r.status_code 200

>>> r.headers['content-type']

'application/json'

>>> r.text

'{"abilities": [ ...

>>> r.json()

{'abilities': [ ... # order may be different

>>> payload = {'key1': 'value1', 'key2': ['value2', 'value3']}

>>> r = requests.get("http://httpbin.org/get", params=payload)

>>> print(r.url) 

http://httpbin.org/get?key1=value1&key2=value2&key2=value3

Test Your API

You can also use unit test

@route('/get/<id:int>'):

def get(id):

return json.dumps(ImageObj)

import unittest

class TestMethods(unittest.TestCase):

def test_get(self):

obj = json.loads(mymodule.get(1)) self.assertEqual(obj['id'], 1)

unittest.main()

file mymodule.py

file test_mymodule.py

Regular Expressions

quentin.bramas@lip6.fr

Regular Expressions

Extract Email Information:

([^@]+)@([^@]+)

[ ]

^

a character that is not

@ the at symbol

+ at least one of this character

>> m = re.match('([^@]+)@([^@]+)',quentin.bramas@lip6.fr)  

>> m.group(1)  'quentin.bramas' 

>> m.group(2) 

'lip6.fr'

(9)

Web Service and Open Data