Displaying Large Amounts of Data (Writing Widgets in Python, Part 4)

May 31, 2003 | Fredrik Lundh

The Widget Construction Kit (WCK) is a programming interface that you can use to create new widgets for Tkinter and other toolkits, in pure Python.

This is the fourth article in a series. In this article, we’ll look at how to implement scrollable widgets.

In this article:

The Scrollbar Interface
Creating a Scrollable Widget
Displaying Huge Data Sets

:::

The Scrollbar Interface #

The WCK uses Tkinter’s scrollbar model, where the scrollbar is a separate widget that can be attached to any widget that supports scrolling.

To bind a scrollbar to the scrolled widget, you set the scrollbar’s command option to point to a method that will be called when the scrollbar is changed, and the yscrollcommand option on the scrollable widget is set to to a method that is called when the view is changed (for example, when new items are added, or the widget is resized).

In the following example, a listbox containing 100 integers is equipped with a scrollbar:

Example: A listbox with a scrollbar

from Tkinter import *

root = Tk()

scrollbar = Scrollbar(root)
scrollbar.pack(side=RIGHT, fill=Y)

listbox = Listbox(root)
listbox.pack()

for i in range(100):
    listbox.insert(END, i)

# bind listbox to scrollbar
listbox.config(yscrollcommand=scrollbar.set)
scrollbar.config(command=listbox.yview)

mainloop()

To watch the traffic between the listbox and the scrollbar, you can replace the Scrollbar and Listbox classes with versions that log the relevant method calls:

class DebugScrollbar(Scrollbar):
    def set(self, *args):
        print "SCROLLBAR SET", args
        Scrollbar.set(self, *args)

class DebugListbox(Listbox):
    def yview(self, *args):
        print "LISTBOX YVIEW", args
        Listbox.yview(self, *args)

scrollbar = DebugScrollbar()
scrollbar.pack(side=RIGHT, fill=Y)

listbox = DebugListbox(yscrollcommand=scrollbar.set)
listbox.pack()

scrollbar.config(command=listbox.yview)

When you run the example using these widgets, you’ll get a stream of SCROLLBAR and LISTBOX messages in the console window.

When the listbox is first displayed, the listbox calls the scrollbar to inform it about the current view (in this example, 10 out of 100 lines are displayed). The scrollbar calls back, informing the listbox that the scrollbar is in its topmost position:

SCROLLBAR SET ('0', '0.1')
LISTBOX YVIEW ('moveto', '0')

Note that all arguments are strings, and that the values are normalized to fit in the 0.0 to 1.0 range.

When you move the scrollbar thumb, the scrollbar sends moveto messages to the listbox. The listbox updates the view, and calls the scrollbar’s set method with the resulting values:

LISTBOX YVIEW ('moveto', '0.1041')
SCROLLBAR SET ('0.1', '0.2')
LISTBOX YVIEW ('moveto', '0.186')
SCROLLBAR SET ('0.19', '0.29')
LISTBOX YVIEW ('moveto', '0.3124')
SCROLLBAR SET ('0.31', '0.41')
LISTBOX YVIEW ('moveto', '0.4166')
SCROLLBAR SET ('0.42', '0.52')

Note that the listbox rounds the scrollbar value to the nearest full line.

If you click outside the scrollbar thumb, the scrollbar generates scroll events.

LISTBOX YVIEW ('scroll', '1', 'pages')
SCROLLBAR SET ('0.5', '0.6')
LISTBOX YVIEW ('scroll', '1', 'pages')
SCROLLBAR SET ('0.58', '0.68')
LISTBOX YVIEW ('scroll', '1', 'units')
SCROLLBAR SET ('0.59', '0.69')
LISTBOX YVIEW ('scroll', '1', 'units')
SCROLLBAR SET ('0.6', '0.7')

For scroll events, the scrollbar provides both a value and a unit, and it’s up to the listbox to interpret the units in a way that makes sense to the user. The value is usually -1 (scroll up/left) or 1 (scroll down/right), and the unit is either pages or units.

In a listbox, the basic unit is usually a single item, and a page is as many items that fit into the widget’s window.

To create your own scrollable widget, you need to deal with two separate issues:

write code that displays a suitable subset of the source data
write code that interacts with the scrollbar, using a Tkinter-style Scrollbar interface

The following example is a list widget that displays a number of strings (stored in a list variable), and allows you to select which one to display at the top of the widget. Use the setfirst method to change the view; use setdata to update the contents.

Example: A scrollable list view

from WCK import Widget, FONT

class ListView(Widget):

    ui_option_width = 20 # in character units
    ui_option_height = 10

    ui_option_font = FONT

    def __init__(self, master, **options):
        self.items = []
        self.first_item = 0 # first visible item
        self.ui_init(master, options)

    def ui_handle_config(self):
        self.font = self.ui_font("black", self.ui_option_font)
        width, self.item_height = self.font.measure()
        return (
            width * int(self.ui_option_width),
            self.item_height * int(self.ui_option_height)
            )

    def ui_handle_repair(self, draw, x0, y0, x1, y1):
        y = 0
        i = self.first_item
        while i < len(self.items) and y < y1:
            draw.text((0, y), self.items[i], self.font)
            y = y + self.item_height
            i = i + 1

    def setfirst(self, first_item):
        self.first_item = first_item
        self.ui_damage()

    def getdata(self):
        return self.items

    def setdata(self, items):
        self.items = items
        self.ui_damage()

#
# try it out

from Tkinter import *

root = Tk()

listbox = ListView(root)
listbox.setdata(map(str, range(100)))
listbox.pack()

listbox.setfirst(10)

mainloop()

To add scrollbar support, you need to add code that calls the scrollbar’s set method (via the yscrollcommand option) whenever the widget’s view is changed. You also need to implement the yview method in a suitable fashion.

The following example adds an update_geometry method which is called whenever the geometry changes. This method notifies the scrollbar, and schedules a widget update.

The setfirst method from the previous example has been extended to make sure that the user cannot move the contents outside the view; without that code, if you scroll to the end, and keep clicking the scollbar arrow, the contents will scroll out of view.

Example: A scrollable list view, with scrollbar support

from WCK import Widget, FONT

class ListView(Widget):

    ui_option_width = 20 # in character units
    ui_option_height = 10

    ui_option_font = FONT

    ui_option_yscrollcommand = None

    def __init__(self, master, **options):
        self.height = 0
        self.items = []
        self.first_item = 0 # first visible item
        self.ui_init(master, options)
        self.update_geometry()

    def ui_handle_config(self):
        self.font = self.ui_font("black", self.ui_option_font)
        width, self.item_height = self.font.measure()
        self.update_geometry()
        return (
            width * int(self.ui_option_width),
            self.item_height * int(self.ui_option_height)
            )

    def ui_handle_resize(self, width, height):
        self.height = height
        self.update_geometry()

    def ui_handle_repair(self, draw, x0, y0, x1, y1):
        y = 0
        i = self.first_item
        while i < len(self.items) and y < y1:
            draw.text((0, y), self.items[i], self.font)
            y = y + self.item_height
            i = i + 1

    def update_geometry(self):
        if callable(self.ui_option_yscrollcommand):
            if self.items and self.height:
                # calculate visible region, in percent
                page_size = self.height / self.item_height
                start = float(self.first_item) / len(self.items)
                end = float(self.first_item + page_size) / len(self.items)
                self.ui_option_yscrollcommand(start, end)
            else:
                self.ui_option_yscrollcommand(0.0, 1.0)
        self.ui_damage()

    def setfirst(self, first):
        # clamp first index
        page_size = self.height / self.item_height
        if first < 0 or len(self.items) <= page_size:
            first = 0
        elif first >= len(self.items) - page_size:
            first = len(self.items)-page_size
        if first != self.first_item:
            # redraw widget
            self.first_item = first
            self.update_geometry()

    def yview(self, event, value, unit=None):
        # adjust top index
        if event == "moveto":
            self.setfirst(int(len(self.items) * float(value) + 0.5))
        elif event == "scroll":
            if unit == "units":
                self.setfirst(self.first_item + int(value))
            elif unit == "pages":
                page_size = self.height / self.item_height
                self.setfirst(self.first_item + int(value) * page_size)

    #
    # list item interface

    def getdata(self):
        return self.items

    def setdata(self, items):
        self.items = items
        self.update_geometry()

#
# try it out

from Tkinter import *

root = Tk()

scrollbar = Scrollbar(root)
scrollbar.pack(side=RIGHT, fill=Y)

listbox = ListView(root, yscrollcommand=scrollbar.set)
listbox.setdata(map(str, range(100)))
listbox.pack()

scrollbar.config(command=listbox.yview)

mainloop()

Displaying Huge Data Sets #

The list view implementation offers a great advantage over the standard Tkinter Listbox, in that it uses a standard Python list, and fetches strings from the list only when it needs them.

If you need to modify the list, all you have to do is to call getdata, modify the object (or replace it), and put it back using setdata:

data = listbox.getdata()
data.sort()
listbox.setdata(data) # triggers a redraw

In contrast, the Tkinter Listbox requires you to transfer data from Python to Tk, and if you want to modify the contents, you have to use Tkinter-specific methods (insert, delete, etc). If you have hundreds or thousands of items, the overhead can be quite noticable.

And if we’re talking millions of items, the Tkinter Listbox will easily gobble up all the memory you have. In contrast, with the list view class, all it takes to display a few million items is a list-like object that responds to the len() function and the [] operator, in the usual way.

Consider this example:

import sys

class huge_list:

    def __len__(self):
        return sys.maxint

    def __getitem__(self, index):
        return str(index)

...

listbox = ListView(root)
listbox.setdata(huge_list())

...

When used with the list view, instances of the huge_list class will behave like a list containing 2147483647 strings (or more, if you’re running it on a 64-bit platform). If you had to create all those strings before displaying any of them, you’d run out of memory on most contemporary platforms, but the list view widget has no problems displaying the entire list:

Note that the huge_list class works pretty much like the built-in xrange method, except that it returns strings instead of integers.

What if we want to use something like xrange right away? The widget expects the list to contain strings, but xrange returns integers. An obvious way to solve this is change the widget so it uses the str function on each item:

class ListView(Widget):

    ...

    def ui_handle_repair(self, draw, x0, y0, x1, y1):
        y = 0
        i = self.first_item
        while i < len(self.items) and y < y1:
            draw.text((0, y), str(self.items[i]), self.font)
            y = y + self.item_height
            i = i + 1

A more flexible solution is to refactor the repair method just slightly, and delegate the drawing to a separate method.

class ListView(Widget):

    ...

    def ui_handle_repair(self, draw, x0, y0, x1, y1):
        y = 0
        i = self.first_item
        while i < len(self.items) and y < y1:
            self.repair_item(draw, (0, y), self.items[i])
            y = y + self.item_height
            i = i + 1

    def repair_item(self, draw, xy, item):
        draw.text(xy, str(item), self.font)

The new repair_item method is called once for each visible item. The default implementation calls str on each item; if you don’t want that, for some reason, you can easily override the method and add your own drawing code.

With this code in place, you can get rid of the huge_list code in the earlier example. Just pass xrange(sys.maxint) to the setdata method, and let the widget take care of the rest.

listbox = ListView(root, yscrollcommand=scrollbar.set)
listbox.setdata(xrange(sys.maxint))
listbox.pack()

Displaying Virtual Data Sets

Displaying insane amounts of integers in a listbox might be a nice way to impress your friends and family at the next user interface toolkit reunion, but it’s probably not something that you will end up doing in a real application. However, you can use the same approach to display more interesting data sets.

In the following example, a simple wrapper class is used as an interface to the result set from a database search. The list view will only fetch enough results to keep the widget up to date:

class SearchResult:

    def __init__(self, database, search_context, number_of_results):
        self.database = database
        self.search_context = search_context
        self.number_of_results = number_of_results

    def __len__(self):
        return self.number_of_results

    def __getitem__(self, index):
        return self.database.getresult(self.search_context, index)

listbox = ListView(root)

context, number_of_results = database.search(query)
listbox.setdata(SearchResult(database, context, number_of_results))

In production code, you should probably add some kind of cache on the way from the database to the display, to avoid fetching the same items over and over again.

Non-Standard Rendering

Since the repair_item hook does all the drawing, it can be also be used to modify the appearance of the list items. Here’s a subclass that takes a list containing (color name, color value) tuples, and draws each name in the corresponding color:

class ColorListView(ListView):

    def repair_item(self, draw, xy, item):
        draw.text(xy, item[0], self.ui_font(item[1], self.ui_option_font))

...

DATA = [
    # CSS1 standard colors
    ("Aqua", "#00ffff"),
    ("Black", "#000000"),
    ("Blue", "#0000ff"),
    ("Fuchsia", "#ff00ff"),
    ("Gray", "#808080"),
    ("Green", "#008000"),
    ("Lime", "#00ff00"),
    ("Maroon", "#800000"),
    ("Navy", "#000080"),
    ("Olive", "#808000"),
    ("Purple", "#800080"),
    ("Red", "#ff0000"),
    ("Silver", "#c0c0c0"),
    ("Teal", "#008080"),
    ("White", "#ffffff"),
    ("Yellow", "#ffff00"),
    ]

...

listbox = ColorListView(root, yscrollcommand=scrollbar.set)
listbox.setdata(DATA)
listbox.pack()

...

And here’s a variant that draws a colored rectangle to the left, and the color name (in black) to the right:

class ColorListView(ListView):

    def repair_item(self, draw, xy, item):
        x0, y0 = xy
        x1 = x0 + self.item_height * 4
        y1 = y0 + self.item_height
        draw.rectangle((x0, y0+2, x1-2, y1-2), self.ui_brush(item[1]))
        draw.text((x1, y0), item[0], self.font)

Ideas for future articles: creating a simple log console, optimizing updates (dirty flags), using the scroll helper mixin, dealing with selections)

back next

Displaying Large Amounts of Data (Writing Widgets in Python, Part 4)

The Scrollbar Interface #

Creating a Scrollable Widget #

Displaying Huge Data Sets #

Displaying Virtual Data Sets

Non-Standard Rendering