The stack:

  • django (non-rel)
  • Google AppEngine
  • djangooappengine

The task:

  • Using Datastore cursors to paginate through a metric bucket load of objects.

Note: if you’re using the webapp framework rather than Django with djangoappengine then this blog post is not for you.

So you’ve got  a model with thousands objects, and you want to display them on a page a few at a time with some lovely pagination.  Django’s standard paginator (django.core.paginator) is a bit dumb, in that it fetches all of the objects and then paginates through them, which other than defeating half the point of paginating, means that this doesn’t really work on App Engine because by default MyModel.objects.all() will only return the first 1000 objects.

So we need to write some custom pagination code.  On AppEngine, MyModel.objects.all()[1001:2000] will work, and will correctly return you objects 1001 to 2000, so we could just use that.  But the beauty of AppEngine is the way that it scales, so we should think big.  When you get to MyModel.objects.all()[100000:100500] the Datastore is going to have to trawl its way through the first 100000 Entities, and then give you the next 500, and that is going to be kind of slow.  This is where Datastore cursors come in.

What are cursors?

A Datastore cursor is just a marker to a starting point in query.  It’s just like giving an OFFSET to a SQL query, but it’s much more efficient because it allows the datastore to jump to the starting point, instead of having to trawl its way through the first <OFFSET> objects before it starts.

Remember: cursors are an AppEngine Datastore thing.  Models are a Django thing.  But djangoappengine has glued the two together for us with 2 functions:

  • get_cursor – give it a Django queryset and it returns the Datastore cursor which marks the position of the end of that query.
  • set_cursor – give it a Django queryset and a cursor and it returns a new queryset which will return its results starting at the object marked by the cursor.

Let’s write some code


#views.py
from djangoappengine.db.utils import get_cursor, set_cursor def my_page(request):
""" View some objects. """
results_per_page = 100
queryset = MyModel.objects.all()
#Note, it hasn't called the datastore yet as querysets are lazy
cursor = request.GET.get('cursor')
if cursor:
queryset = set_cursor(queryset, cursor)
results =queryset[0:results_per_page] #starts at the offset marked by the cursor
cursor_for_next_page = get_cursor(results)
template_vars = {
"results": results,
"cursor_for_next_page": cursor_for_next_page,
}
return render_to_response("template.html", template_vars)


#template.html
<h1>Page 1</h1>
<a href="/myview/?cursor={{cursor_for_next_page}}">Next page</a>
{% for object in page1 %}
{{object}} {#whatever you're displaying #}
{% endfor %}

This is fine when you just want a ‘next’ link, but what if you want to be able to skip to page 5 or page 10? Then it gets a bit more tricky.

It’s time for me to eat some dinner, but if this post gets some comments then I’ll expand on this stuff a bit more and reveal ways of making pagination links to other pages (including the previous page).

4 Responses to “Pagination with App Engine cursors in django-nonrel”

  1. Sam says:

    Can you expand on this please? I’m really interested to know how you would implement pagination using cursor with page links. Thanks in advance.

  2. saint says:

    nice post!.
    whenever you are done with your dinner, it would be good if you could give us some clues about how to do pagination in large querysets after filtering and ordering (more than 1000 entities), skip pages and get total count of pages without fetching everything.

  3. wasert says:

    Great article, thanks.

    I am using it to display results from a full text search and it is very similar. I have found it not so simple to include the previous page’s cursor. I would be greate if you could write another post showing how to include the previous page.

    Thanks again

  4. admin says:

    Requests for more info acknowledged! I’ll try to find some time to write part 2 of this in the coming week.

Leave a Reply