APRIL 22, 2010

Python urllib2 timeout issue

I use urllib2 from Python's standard library, in quite a few projects. It's quite nice, but the documentation isn't very comprehensive and it always makes me feel like I'm programming Java once I want to do something more complicated than just open an URL and read the response (i.e. handling redirect responses, reading response headers, etc).

Anyway, the other day I found - if not a bug - then at least an undocumented issue. Since Python 2.6, urllib2 provides a way to set the timeout time, like in the following code where the timeout is set to 2.5 seconds:

import urllib2

try:
    response = urllib2.urlopen("http://google.com", None, 2.5)
except URLError, e:
    print "Oops, timed out?"

If no timeout is specified, the global socket timeout value will be used, which by default is infinite.

The above code will catch almost every timeout, but the problem is that you might still get a timeout raised as a totally different exception:

File "/usr/lib/python2.4/socket.py", line 285, in read
  data = self._sock.recv(recv_size)
File "/usr/lib/python2.4/httplib.py", line 460, in read
  return self._read_chunked(amt)
File "/usr/lib/python2.4/httplib.py", line 495, in _read_chunked
  line = self.fp.readline()
File "/usr/lib/python2.4/socket.py", line 325, in readline
  data = recv(1)
socket.timeout: timed out

The solution is to catch this other exception, thrown by python's socket lib, as well:

import urllib2
import socket

try:
    response = urllib2.urlopen("http://google.com", None, 2.5)
except URLError, e:
    print "Oops, timed out?"
except socket.timeout:
    print "Timed out!"

Hopefully this will save someone else some headache :).

Comments

MARCH 03, 2010

Favorite Django Tips

A few months ago I found a really useful Stack Overflow Question. Here are my favorites from the answers.

Use render_to decorator instead of render_to_response

This decorator is found in the app django annoying, and is a very nice shortcut for declaring what template a view should render.

Instead of returning the response of render_to_response, you just return a python dict which will be used as the template context for the template specified as argument to the @render_to decorator. If anything else than a dict is returned, normal view processing will occur, so this won't break redirects or any other cases where you might return a HttpResponse (for example normal render_to_response code).

Anyway, here is an example on how to use it:

@render_to("list.html")
def list_posts(request):
    posts = BlogPost.objects.all()
    return {"blog_posts": posts}

This equals to:

def list_posts(request):
    posts = BlogPost.objects.all()
    return render_to_response('list.html',
        {'blog_posts': posts},
        context_instance=RequestContext(request))

Update (22/4): Marcin Nowak notified me that the render_to decorator breaks Django Reusable App converions, so I made a fork of django-annoying where I modified the render_to decorator to support template_name and extra_context keyword arguments.

Load custom template tags in all templates

Custom template tags that you use all over your templates can be auto loaded. Just add the following in a module that is loaded (i.e. your urlconf if you want the template tags to be loaded for the whole project)

from django import template
template.add_to_builtins('project.app.templatetags.custom_tag_module')

Use relative paths in settings.py

I hesitated about adding this tips, since I think it's quite obvious, but since so many people on Stack Overflow has voted it up, I guess there are people who use(d) absolute paths in their settings.py.

Don't use absolute paths in settings.py (i.e /home/jonatan/...), instead use relative paths so that the project will work wherever it resides.

import os.path
TEMPLATE_DIRS = (
    os.path.join(os.path.dirname(__file__), "templates"),
)

Comments