Concept for an Image Processing App

This is a writeup for a concept I had been working on – an image processing application. It did not come to pass but I’d like to share some of the ideas behind it (previous posts touches upon some ideas too). It might be of interest to those of you working on developing full-scale applications integrated in web browsers.

The early days

An early sketchup for the application took inspiration from audio-processing software. In particular SynC Modular (later known as Reaktor). It had several levels of building instruments and effects for audio processing. At the high level, non-expert users could wire up high-level modules that had well-defined input/output sockets. In addition, UI for parameters could be designed that allowed users to configure the internals of modules. At the low level, expert users could program up logic for signal manipulation using predefined low-level primitives. The expert users could then package up the internals in a high-level module that exposed inputs/outputs for wiring.

In my first iteration, the user-interface consisted mostly of html/javascript/css code, but resorted to Flash for display of image, graph manipulation, and processing. Since Flash must be compiled before execution, image processing had to use predefined modules when deployed. New modules resulted in a re-compilation and deployment of the application. This restriction would hinder the usability of the application so the search for an alternative started. The only alternative present at the time was the Java VM. It’s usage was very low but still widely deployed. Next was the search for a compiler that could allow runtime compilation. I stumbled upon the Scala Programming Language which included an open-source compiler. Besides solving my problem, the language was also a great teaching tool for programming languages. With the Scala compiler at hand I could migrate the Flash-based processing to a Scala-based applet.

Putting the ideas into action

Here is a small picture showing the general layout of the application. It shows the result of mandelbrot fractal applied to an image. The module generating the fractal has several input parameters and an input image to apply it to, then an output image to store the resulting image to.

When editing the Fractal node we can see the actual Scala code that generates the image. In this code, the input image is transformed by mapping each pixel over a lambda function which yields the transformed pixel. Behind the scenes, a new image is produced, then stored as the output image.

Pixels in images are not manipulated directly with xy-coordinates. This allows the application to subdivide the image into segments that can be transformed concurrently. In addition, an initial pass on a miniaturized image is performed, allowing the user to view a quick preview. The final image is then progressively refined in the last pass.

Leave a Reply

Your email address will not be published. Required fields are marked *