Python 3.0 AmazonS3 with Proxy Authentication

I just choose Python to do prototyping for Amazon S3. I just used 3 lines of http client module and viola, done with uploading. This is the same for URL Query String authentication. Just needs to know how Amazon wants their signature in the URL. Python almost built everything in.

I hit a wall, my POC whizz-banged when I need to fiercely encounter proxy. Devil is details. Yes, proxy needs to be authenticated. Since I used Python 3.0 I never able to find a suitable open source module to elegantly handle Proxy Authentication. I tried with socket and used CONNECT method pass through proxy but some how I was not able to send a proper HTTP bits past proxy. It looked like somewhere malformed.

I again went back to Url-lib with user id and password passed as part of the url itself (http://USER_ID:PASSWORD@http_proxy_url:PORT).

Below method prepares the proxy,

self.proxyURL = http://USER_ID:PASSWORD@http_proxy_url:PORT

def prepareProxy (self) :

proxy_handler = urllib.request.ProxyHandler({'http': self.proxyURL})

proxy_auth_handler = urllib.request.HTTPBasicAuthHandler()

opener = urllib.request.build_opener(proxy_handler, proxy_auth_handler)

urllib.request.install_opener(opener)

return opener

Then, opener can be used to fire a http request.

Another issue is, url opener is not http1.1 complaint. It didn't allow me to use HTTP PUT request. After a long goolging I found, it is possible by extending Request object,

I used something similar as below,

class PutRequest(urllib.request.Request):

def __init__(self, url, data=None, headers={}, origin_req_host=None, unverifiable=False):

self._method = 'PUT'

super().__init__(url, data, headers,

origin_req_host, unverifiable)

def get_method(self):

return self._method

All the 2 parts baked together, I am able to use python3.0.

Note: Amazon has its own python API but it didn't allow me to upload in blocks. It is current implementation needs to read the entire file and send it as binary data which is very in-efficient and not possible for bulk data.