The stack:
- django (non-rel)
- Google AppEngine
- djangooappengine
The task:
- Using Datastore cursors to paginate through a metric bucket load of objects.
Note: if you’re using the webapp framework rather than Django with djangoappengine then this blog post is not for you.
What are cursors?
A Datastore cursor is just a marker to a starting point in query. It’s just like giving an OFFSET to a SQL query, but it’s much more efficient because it allows the datastore to jump to the starting point, instead of having to trawl its way through the first <OFFSET> objects before it starts.
Remember: cursors are an AppEngine Datastore thing. Models are a Django thing. But djangoappengine has glued the two together for us with 2 functions:
- get_cursor – give it a Django queryset and it returns the Datastore cursor which marks the position of the end of that query.
- set_cursor – give it a Django queryset and a cursor and it returns a new queryset which will return its results starting at the object marked by the cursor.
Let’s write some code
#views.py
from djangoappengine.db.utils import get_cursor, set_cursor
def my_page(request):
""" View some objects. """
results_per_page = 100
queryset = MyModel.objects.all()
#Note, it hasn't called the datastore yet as querysets are lazy
cursor = request.GET.get('cursor')
if cursor:
queryset = set_cursor(queryset, cursor)
results =queryset[0:results_per_page] #starts at the offset marked by the cursor
cursor_for_next_page = get_cursor(results)
template_vars = {
"results": results,
"cursor_for_next_page": cursor_for_next_page,
}
return render_to_response("template.html", template_vars)
#template.html
<h1>Page 1</h1>
<a href="/myview/?cursor={{cursor_for_next_page}}">Next page</a>
{% for object in page1 %}
{{object}} {#whatever you're displaying #}
{% endfor %}
This is fine when you just want a ‘next’ link, but what if you want to be able to skip to page 5 or page 10? Then it gets a bit more tricky.
It’s time for me to eat some dinner, but if this post gets some comments then I’ll expand on this stuff a bit more and reveal ways of making pagination links to other pages (including the previous page).


Can you expand on this please? I’m really interested to know how you would implement pagination using cursor with page links. Thanks in advance.
nice post!.
whenever you are done with your dinner, it would be good if you could give us some clues about how to do pagination in large querysets after filtering and ordering (more than 1000 entities), skip pages and get total count of pages without fetching everything.
Great article, thanks.
I am using it to display results from a full text search and it is very similar. I have found it not so simple to include the previous page’s cursor. I would be greate if you could write another post showing how to include the previous page.
Thanks again
Requests for more info acknowledged! I’ll try to find some time to write part 2 of this in the coming week.