Welcome to blogs.conchango.com

Few months ago I have been researching with some colleagues new ways of visualizing data. Infovis has been a theme I've always been interested in because it is a field where rich interactions can be of real help on a functional level. The most worthy piece I did in the past is probably Tagged Colors. One of the aims of the research was to that the charts should be able to manage time browsing, so I thought straight away how common charts could handle it. I am not a big fan of pie charts but when imagining it in a 3D environment, it got more interesting. Of course I had to show to the others what I had in mind and for this reason I picked a tool I was pretty familiar with, Processsing. Processing has been created for rapid prototyping and it is especially useful if you need simple drawing API (lines, points, polygons), of course you can do much more complex things with it.

Below you can see the result after few hours with Processing and Eclipse.

Time based Pie and Bar Chart from nuthinking on Vimeo.

As mentioned on my previous post, my next experiment would have been connecting the iPhone to a Flash application. So here it is:

iPhone 2 Flash from nuthinking on Vimeo.

Design details

I managed to connect to the iPhone a couple of weeks ago but I wanted to create a decent demo to better show the technical achievement. Multi-touch interactions have still a long way to go and so far it's quite hard to see it useful for contexts different than editing, and of course multi-user experiences. In this experiment the user is able just to modify size and orientation of a carousel and its scrollbar, nothing more a part of course being able to scroll the carousel. I'm definitely interested on seeing how multi-touch interactions will be applied to standard UI components, as I showed also with my previous experiments.

Technical details

To create a bridge from iPhone to a Flash application I needed to use an iPhone application which would connect to a socket server the Flash application would receive data from. After I could try the efficiency of Multitouch Framework, I used their iPhone application and their framework for the Mac application, then I imported some classes from OpenFrameworks' addons to create the socket server (the same I used on one of my previous experiment), that's it. I would like to share though the most interesting parts of the process.

As you can see, after the finger starts to move a delay starts growing. This is because at any finger movement the iPhone sends a package of information to the application via network, and these packages are sent very frequently so that the network starts accumulating them. To solve a problem like this, you should avoid to send information about small movements. I implemented this on my mac application, but unfortunately the problem seems to be in the root, the iPhone application. The iPhone application seems to send to their framework all the information, without filtering them, the framework, in facts, returns all the events in basically the same way you would get them in an iPhone application, it would be nice to have a tolerance, which you can modify, for small movements so that you can optimize the communication, as I did on my Mac app. The only annoying part of this is that you have to identify the finger, thing that doesn't happen by default with iPhone SDK, but it's very easy to do by the way.

Other thing I wanted to mention is that despite the idea of using Multitouch Framework was to test multi-touch applications quickly, thus just using the iPhone as input device, I had to find an even more portable way to test it. In facts the Mac I usually use to write ActionScript doesn't have Leopard and thus is not the one I use for Mac development (with Objective-C 2.0), so I wanted to be able to test it without anything else than the usual input information a Flash application can give. The solution was pretty easy and combined perfectly with the controllers I was writing. I emulated the finger press just double clicking with the mouse, this would create a touch view which I can then drag or removing with another double click. Precisely the application can easily switch between the socket multi-touch controller, to the mouse emulated one. Here is a capture of this modality:

Multitouch testing in Flash from nuthinking on Vimeo.

When I started to test the iPhone as input device for network connected applications I saw instantly as obstacle the fact that the iphone can only track 5 fingers and that its surface is very small, comparing to common multi-touch surfaces. At the moment my emulated fingers can be dragged only one at the time, but it should be pretty easy to group some of them to move them together, in fact, as shown on my previous experiments, some interactions might required it and so it could be pretty handy to add this feature.

What's next? Multitouch framework is evolving quite rapidly, so I can't exclude I will experiment more with it soon, but at the moment I'm trying to be focused on an iPhone game, tilt based, I started designing and developing.

Stay tuned then!

Here a small tip which makes sense but wasn't necessary obvious. You can have your C++ class both as .h file plus .cpp file but also as .h file plush .mm file. The important part is that the Objective-C class which includes it NEEDS to have the .mm extension, not the default .m. Otherwise you get C++ header syntax errors in the include line in the .m file.

So after having your Objective-C class renamed properly (with .mm extension) you can include your C++ class and using it with the usual C++ syntax.

Give the Foo class:

#ifndef FOO_H_INCLUDED
#define FOO_H_INCLUDED

class Foo
    {
    public:
        void setId(int newId);
        int getId();
    private:
        int myId;
    };

#endif // FOO_H_INCLUDED

You can use it like this:

Foo *foo = new Foo;
foo->setId(16);
NSLog(@"foo id: %i", foo->getId()); // foo id: 16
delete foo;

I am still thinking how much to use Objective-C and how much C++ in my future projects, especially for libraries, using C++ might mean more reusable, accessible, and probably readable, code but I found in the Objective-C dynamism a powerfull tool which can allow me to implement easily design patterns which I use regularly in the ActionScript world.

Like many people, at least that I know, I am having a better look to the iPhone SDK. Not only because I have an iPhone and an iPod-Touch, but also because I was curious to see which is the Apple way on building applications, especially on mobile devices. In the past I made some experiments with J2ME, PalmOS and Flash Lite; each one with its own pros and cons. I am still on early stages to give a deep opinion about the language Apple uses (Objective-C) and their framework. But so far I have to admit I have been positively impressed. Because NDA I don't think I can say much else about this.

By the way, iPhone brought to us, user interface designers, new potentialities and of course new issues. I am not suprised that features like multi-touch, which is in a very hype moment recently (thanks also to Microsoft Surface and Co.), are used in small and isolated cases. New paradigms need to be made but, especially considering the "natural" approach that Apple had when designing iPhone OS, it gets tricky to decide which will be best.

So in these days where people are trying to understand how those kind of interfaces should behave, I took the liberty to experiment with the scrolling of entire screens as a power user like me could find easier. I know that having two different ways of doing the same thing is not ideal, but first time I saw how to scroll between the previous and the next photo, for instance, I hoped it was a joke. I am still far from the truth but this doesn't mean of course that the current UI behaviors cannot improve using different approaches.

From the video that will follow, you will be able to notice three modalities which can co-exist:

Normal
It is basically how the iPhone does currently, to scroll from a screen to another, you just have to drag it for more of than the half of the screen, in few words the screen that will be seen the most after releasing will be the next one.

Continuous
Why on earth the dragging should have inertia? Without it, if I wanted to drag it a bit I could just drag it slower. After releasing I leave the eventual speed the user set until the screens end.

Power
When two fingers are on the screen, they work as scrollbar thumb. This is a bit a tricky one, especially when you have a long list of screens (50 is probably a discreet number), because can be hard to use. It is also true that some easing could be applied to give a more smoother control, but less reactive. With this method you can scroll from the first to the last one at the speed of light.

It is clear that having 50 screens available at "the speed of light" can bring, especially on mobile devices, ram problems because they should be loaded at the start, but this is just an experiment and these behaviors could be easily apply in less demanding contexts.

It has been fun to play with the multi-touch and see how Apple framework handles them and I am looking forward to see which will be the behaviors that will become standard, at the moment let's enjoy the experiments!

You might ask: “how on Earth computer-vision is related to RIA?”, my answer is “as Microsoft Surface is”.

Computer-vision is the technique used to make machines understand the physical environment through images. Used of course a lot by robots but getting more and more common, because of its affordability, as “input device” for computers allowing a more natural interaction (without mouse or keyboard) with the software. The recent boom of computer-vision is in facts due to multi-touch interfaces, like the ones built by Jeff Han and later by Microsoft (Surface and Touch Wall), which are making many users dreaming one day to browse their files like Tom Cruise in Minority Report.

With Surface, Microsoft is trying to create a solution where the computer not only tracks fingers but also items, this probably happens through fiducials presumably just added to the items as sticker (thing easily achievable with reacTIVision).

A big advantage of using cameras as input and projectors as output is indeed that your interactive surface is very scalable and the size is basically limited only by the projection/camera distance and angle.

The images that are retrieved by the camera have to be analyzed depending on the information we want to read from them. External factors like lighting or background can make the process very hard and having a good setup, when possible, is crucial. To analyze the images, general image processing algorithms have to be used, from blur or median filters to dilation and erosion, which can be very useful for instance when looking for connected components (here some fundamentals of Image Processing).

Luckily there are already libraries that hide many complex algorithms from you, but it is good to know how they work and how to better prepare the images for them.
THE C++ library is OpenCV, in the java world JMyron is quite popular.
With higher level front-ends, like Flash, the most common approach is to use a proxy application which analyzes the images and sends the results through socket. An example could be Touchlib.
Said that, since Flash can manage one webcam and allows you to perform low level image processing, it can be already used to create a complete solution, from input to output, by itself and I’m sure BitmapBlogDetection after a couple of bitmap filters can help you doing a multi-touch system in no time, and most important only ActionScript!

For further readings I would recommend to follow my related tag on del.icio.us.

Flash applications have always been, thanks to classes like XMLSocket, a good client for push http services. On the web you can find many chat applications, on the client, instead, it's quite common to connect the application to a local proxy server. This proxy servers usually create an http socket on localhost and send to the Flash clients via TCP any kind of data as string. On ActionScript3 the Socket class, and some others, can read directly byte arrays and it becomes really easy to send optimized data packages to the Flash client. Let see an example!

I was playing with a small application in C++ built with openframeworks, that sends to the Flash client as package a model that contains mouse coordinates and the distance to the screen. Think about a touch-less interface sensor layer, for instance. The C++ class would be:

 
// model.h

#ifndef _MODEL
#define _MODEL

class Model
{
    public:
        Model(void);
        ~Model(void);
       
        int x;
        int y;
        float depth;
};

#endif

This class specifies its attributes, and most important, their order.

So if I send an instance via socket using OF (openframework) ofxTCPServer :

ofxTCPServer.sendRawBytesToAll((char*) modelInstance, sizeof(Model));

In Flash I can easily retrieve the values in this way:

var x : int = mySocket.readInt();
var y : int = mySocket.readInt();
var d : Number = mySocket.readFloat();

The package sent each time is 4+4+4 bytes and is sent pretty quickly :)

An abstract class that fills itself based on its attributes could also be made. Here a very quick draft:

// VariableType.as

package com.conchango
{
    public class VariableType
    {
        public static const INT : VariableType = new VariableType("int",4,"readInt");
        public static const NUMBER : VariableType = new VariableType("Number",4,"readFloat");
       
        public var name : String;
        public var size : int;
        public var readMethod : String;
       
        private static var map : Object;
       
        public function VariableType (name : String, size : int, readMethod : String)
        {
            this.name = name;
            this.size = size;
            this.readMethod = readMethod;
           
            register(this);
        }
       
        private static function register (type : VariableType) : void
        {
            if(map == null)
                map = {};
            map[type.name] = type;
        }
       
        public static function getTypeByName (name : String) : VariableType
        {
            return map[name] as VariableType;
        }
    }
}

// Streamable.as

package com.conchango
{
    import flash.utils.getQualifiedClassName;
    import flash.utils.IDataInput;
   
    public class Streamable
    {
        public function Streamable (input : IDataInput)
        {
            var attribute : String;
            var typeName : String;
            var type : VariableType;
            for each(attribute in attributesList)
            {
                typeName = getQualifiedClassName(this[attribute]);
                type = VariableType.getTypeByName(typeName);
                this[attribute] = input[type.readMethod]();
            }
        }
       
        protected function get attributesList () : Array
        {
            return [];
        }
       
        public function get size () : int
        {
            var res : uint;
            var attribute : String;
            var typeName : String;
            var type : VariableType;
            for each(attribute in attributesList)
            {
                typeName = getQualifiedClassName(this[attribute]);
                type = VariableType.getTypeByName(typeName);
                res += type.size;
            }
            return res;
        }
    }
}

// Model.as

package com.conchango
{
    import flash.utils.IDataInput;
   
    public class Model extends Streamable
    {
        public var x : int;
        public var y : int;
        public var depth : Number;
       
        public function Model(input : IDataInput){
            super(input);
        }
       
        override protected function get attributesList():Array
        {
            return ["x","y","depth"];
        }
    }
}


Really cool thing would be to send serialized objects to Flash using AMF3 protocol, but unfortunately I couldn't find at the moment any porting of it for C++, I might try with Java soon though.

Other interesting aspect is that Flash Player 10 will support UDP, as OF does, but unfortunately its peculiarity of not check every packet won't be make it suitable for any application.