Tag Archives: ui

Concept for an Image Processing App

This is a writeup for a concept I had been working on – an image processing application. It did not come to pass but I’d like to share some of the ideas behind it (previous posts touches upon some ideas too). It might be of interest to those of you working on developing full-scale applications integrated in web browsers.

The early days

An early sketchup for the application took inspiration from audio-processing software. In particular SynC Modular (later known as Reaktor). It had several levels of building instruments and effects for audio processing. At the high level, non-expert users could wire up high-level modules that had well-defined input/output sockets. In addition, UI for parameters could be designed that allowed users to configure the internals of modules. At the low level, expert users could program up logic for signal manipulation using predefined low-level primitives. The expert users could then package up the internals in a high-level module that exposed inputs/outputs for wiring.

In my first iteration, the user-interface consisted mostly of html/javascript/css code, but resorted to Flash for display of image, graph manipulation, and processing. Since Flash must be compiled before execution, image processing had to use predefined modules when deployed. New modules resulted in a re-compilation and deployment of the application. This restriction would hinder the usability of the application so the search for an alternative started. The only alternative present at the time was the Java VM. It’s usage was very low but still widely deployed. Next was the search for a compiler that could allow runtime compilation. I stumbled upon the Scala Programming Language which included an open-source compiler. Besides solving my problem, the language was also a great teaching tool for programming languages. With the Scala compiler at hand I could migrate the Flash-based processing to a Scala-based applet.

Putting the ideas into action

Here is a small picture showing the general layout of the application. It shows the result of mandelbrot fractal applied to an image. The module generating the fractal has several input parameters and an input image to apply it to, then an output image to store the resulting image to.

When editing the Fractal node we can see the actual Scala code that generates the image. In this code, the input image is transformed by mapping each pixel over a lambda function which yields the transformed pixel. Behind the scenes, a new image is produced, then stored as the output image.

Pixels in images are not manipulated directly with xy-coordinates. This allows the application to subdivide the image into segments that can be transformed concurrently. In addition, an initial pass on a miniaturized image is performed, allowing the user to view a quick preview. The final image is then progressively refined in the last pass.

A compact UI layout for tag-based search

I’ve always been impressed by tags as a way to do search and have been looking to make use of them myself. In my application my initial design for categorizing items was based on the familiar directory-structure, but this presented several problems in relation to search. I needed a more efficient way of locating items so I chose to replace the old solution with a tag-based one. An added bonus was that I could reuse this several other places in my application.

In my application the screen space available for tag search is fairly small. This led me to experiment with how of represent the UI elements. A current sketch is given below. It consists of a tags- and values-section. In the tags-section the left-hand side gives a search input and a page browser. The right-hand side lists the current page of matching tags (constrained by the search text and selected tags). In the values-section a similar search input and page browser is found. The right-hand side lists the current page of matching values (constrained by the search text and selected tags).

In the previous sketch no tags were selected. The next one shows when two tags has been selected, where tag4 is included and tag7 is excluded. Together these will constrain which tags and values are currently searchable.

A possible extension could be types of tags. Not sure if this is something I’d like to add but could be useful for integrating other metadata into the search system. Here tags are extended with types and search input extended with a type constraint which can also reuse the search mechanism for selecting a different type. To abbreviate tag naming some sort of color coding could for example be used to represent types.

In summary, I’m still experimenting with this layout but it seem to working as intended. In my case the search will probably involve some 10 to 1000 items. Tags are usually employed on much larger volumes but I think this UI layout will work out atleast for my setup.

Deep Pocket Layout – tree-based UI layout

Overall idea

The Deep Pocket Layout is a specialized tree data-structure for use in non-overlapping layout of widgets. This data-structure can be used for partitioning screen-space, like the quadtree or kd-tree data-structures, and uses the naming conventions of border layout. In usage, it functions as a tree of either vertically or horizontally linked partitions. Much like a combined border/horizontal/vertical layout.

This data-structure will be explained by example as a container widget, which is part of an inheritance hierarchy of widgets.

Introduction by example

The figure above shows the named “pockets” of the data structure. The outmost pockets are used for linking the Pocket Layout Widget (PW) together with other PWs. The center pocket can contain a PW (thus forming a tree) or a different kind of Widget (leaf node). The center pocket specifies the screen-space, while the border pockets acts as internal links to other PWs. When the left or right pockets are linked, they become part of a horizontal layout (top and bottom pockets must always be empty). Conversely, when the top or bottom pockets are linked, they become part of a vertical layout (left and right pockets must always be empty).

The first example below shows how a widget (of any kind) can be inserted by using the center pocket of the PW.

PWs can be arranged into either vertical and horizontal layouts. The figure below shows how a horizontal layout is started. Since P1 it the root of the tree, a new PW P3 is created, replacing the center pocket content of P1. Then P2 is linked with P3, starting a horizontal layout.

The figure below shows how a vertical layout is started. This example is identical to the previous one, but starting a vertical layout instead. (The root constraint for P1 still applies.)

The final example shows how to split a PW in a horizontal layout into a vertical. Here P3 is already part of a horizontal layout. A new PW is created and replaced with P3s center pocket. The vertical layout is then started by linking P6 with P7.

The final deep pocket layout of this example is visualized in the following figure.


Some features:

  • Enables combination of tree-based vertical/horizontal layouts.
  • Allows the user to insert/remove any PWs in the hierarchy from any depth, thus controlling the complete layout with drag-and-drop.
  • User control over screen space that is delegated within the tree hierarchy.
  • PW linked in either a horizontal or vertical layout can adjust neighboring PWs or delegate request to parent PW.

Implementation extensions:

  • The example used “share equally”, but custom ratios of the linked PWs can be handled by the parent PW.
  • Overlay of drag-and-drop targets when dragging is in progress.