HCI Research & Tech.

Input Pipelines and Sensors Flow Soup

2011-12-13T17:38:00.000+01:00

Fig 1. Input Layer Abstraction

By convention, Input devices are the peripherals which detect a human user input. And sensors are those which detect the environmental variation, related or not to the human.

Despite the existence of other definitions, both Input devices and sensors are needed to provide rich interaction with the user and allowing him to accomplish the tasks he want in less time.

Input devices can be the keyboard and the mouse and almost every application supports them but the definition also include touch and multitouch screens, touchpad, joysticks, etc.

From a wider point of view, keyboard and mouse seems to be two constant: K1, K2. They are modeled in operating systems using Finite State Machines and they are routed to the application without too much alteration of their original content. (Except the transformation of raw codes to ascii or unicode ones for example)

The "Why"

Operating systems, and I'll take Linux as an example, have abstraction layers to handle the input. Every peripherals generating key clicks is seen as a keyboard. For touchpads, mice, touchscreens, pen, etc they are attached to a virtual pointer device, and are handled as a conventional mouse.

Device drivers may generate other information (finger blob size, blob orientation, etc.) but they are all eliminated and don't reach the application.

Last years, we have faced the emergence of an 1.unlimited number of input devices and sensors, each of them provide very different information. And in the other side, 2.rich application, exposing a lot of features, or which have more dimensional variables than ones found in a single input device (3d object control in 3d environment).

With these 2 poles, the problem becomes obvious:

Why is the operating system preventing us from using all devices capabilities inside our rich applications ?

Ok, let's fix this and answer the question with another :

Do you have another solution for mapping input events to application features ?

Mapping Input events

Before speaking about mapping, we should study the input itself even taking few examples.

Touchpads have their input considered as a compatible form of mouse input. But their brute information comes very noisy. When using direct input your pointer moves in a similar way of a Parkinson disease patient. The developers of X.org have added a "filter" which smooths the movement (named "Response-augmented Exponentially Weighted Moving Average Filter").

Speaking about mouse input, the X.org add also another "filter" that alter the smoothed input to accelerate its movement, so you a user don't need to slide a lot of times to move the pointer in bigger screens.

And let's imagine that this flow represented in Fig 2. finishes as an input to control the camera in a 3D scene:

Fig 2. Input Flow to control a 3D scene

From the last representation, the input system can be seen as a flow and in each point, the input got transformed before being routed to the application. The 3D scene itself takes that input and maps it internally to control the camera view.

The problem in this case is that, what if you want that your management meets the standard and won't be considered as a hack? You can't avoid the virtual pointer, and you can't avoid this predefined set of filters. You can't also select the manner of which the input can be mapped inside the application ( mouse(x,y) → scene(x,y) or → scene(y,z) etc.)

Another problem is that, some filters are inside the input device driver, other are inside the Input layer, and the rest belongs to the application. And all of them are hardly written and can't be changed or remapped without recompilation of the full stack.

Any time we have a new device or new events to support in the system, we keep asking this question:

How we will fit the new device input inside this flow with minimum effort and the least loss of information ?

Rethinking All the Stack
In 2009, I have discovered all of this and I have found some efforts to simplify input management. Just having the ability to imagine a possible change is a lot of bravery... For a system developer who only use a keyboard, why he should think about rewriting everything ? That's a lot of work man !

From the last figures, we can imagine that filters should be extracted outside of device drivers, and from any predefined flow in a system, and we can play with them to modify and transform raw input to any useful form.

The flow becomes a general graph, where a sensor input can control how much a filter transforms a device stream to another. For real-time and other needs we may add also a universal timing to master how much each filter takes time for his computation.

Fig 3. Input Flow as represented by StreamInput Khronos Model.

What about applications ?
Legacy applications takes only two sort of input : Keyboard and Mouse. These two inputs still very limited to what is possible to do. But We have already have filters which route information through them including keyboard mapping transformers (possibility to write in Arabic/French/Chinese with the same keyboard), mutlitouch injectors in legacy apps like Ginn (included in ubuntu), or whatever else.

But wait, shouldn't we rethink the application itself ? Why they don't expose their functionalities through a Software Bus, where we have the freedom to connect filters' input to the specific action to be performed?

Fig 4. Rethinking application input

By doing this the biggest work will become the mapping. And we will need to search best ways to do it to perform user goals in better ways.

Scientific background ?

A lot of scientific research have studied these problems. But each of them tried to fix either the input flow, or the best mapping. But not all the cycle from and to the user.

For the Input configuration you can see the work of Pierre Dragicevic : iCon.

Another interesting thesis is the one by Rami Ajaj (PhD in French)

Theoretical and architectural support for input device adaptation paper.

A lot of other notable exist but I can't cite it all.

And for a deep study of mapping and to understand the need of filters, it is mandatory to study what is an input device, a sensor, do some morphological analysis of their design space, study actual standards implemented in operating systems and describing a large spectrum of devices and usages.

(I have skipped many research areas related to this subject to keep this post clear and simple.)

Knowledge Grows When Shared

2011-12-07T17:29:00.001+01:00

Long Time !

Many things happened and I haven't blogged about them !
Let's see if I can fill the gap...

Reviewing the Google Chrome

2010-01-16T15:16:00.002+01:00

Google has invested a huge quantity of money and marketing for its Chrome browser. Sometimes people install it then use it without knowing that they changed their browser, by having only followed the link in the top right.

People install Chrome by clicking the link in Google page.

Many computer engineers know that this navigator has many advantages and technological power as being the fastest browser (for its Javascript interpreter/PseudoCompiler, and also from a usability PoV), the use of the best html/CSS engine (webkit), the SKIA library for showing graphics (few people know about this library now being called/integrated as chromium, firefox itself use cairo for rendering).

So, after all of this, is it only a bunch of technology assembled in disorder ?
From my point of view, and after having some human computer interaction courses, I started to view things in a different ways. And I have detected these points :

The application of Fitts Law

Tabs are placed in the top of the screen, so the virtual growth of buttons are bigger, and we can switch tabs a lot easier than the traditional way.

(Apple use the same law from ages having always the menu in the top of the screen)

Reducing human errors

People can make errors easily when trying to switch from one tab to another, when the width of the tab is too small, the probability of making the mistake of closing the tab rather than switching to it.

Google removes the close button when the tabs width is too small.

Better feedback

I really enjoyed the search function, when I search Google tries to indicate the word positions inside the document but also in the right and exactly in the sliding bar. Every small yellow line indicates the word position.

Better use of the screen

In Chrome, all the screen is used, no status bars, when there is a need to show something, it is shown using the needed place.

Better interaction :

When closing a tab, the next close button comes to the current mouse cursor to facilitate closing others. (closing tab is contagious ?)

The tabs are redrawn only when you move your mouse outside the tabs.

Being different form Firefox, Chrome have only one place to enter urls, text search and every things.

The browser is intelligent enough to know what the user wants.

Better usability :

In Chrome, every text input is resizable. When it is not wide as you need, just make it bigger.

Many other details exist (I speak about "Chrome only" features) and needs to be cited here, but these are the most important one.

This doesn't mean that Chrome is perfect. It has some problems dealing with RTL texts like Arabic sites that aren't shown correctly.

And the recovering of a crashed/closed browser (it exists but I feel it very bad). Some times I can't see the recovering button but in Firefox it is always present after a crash.

Visualization of Algorithms, how should it be ?

2010-01-16T14:02:00.000+01:00

Last days, while I was teaching a student some sorting algorithms. I asked myself how can I explain it the easiest way. The evident solution was to use some visualization tips.
OK OK, I say some "InfoViz", but then, the big question would stay the same : HOW ?

According to the human-computer interaction course I have got, Humans don't react like machine, they should have "Mental Models" to manipulate information and resolve problems. We don't have one Human mental model but we are sure about some reactions.
According to (Philip Johnson-Laird, 1983), humans need examples to how a mathematical functions work (hmmm, we don't speak about math nerds here...), so when we see a formula, we need to play it on some examples in order to construct our mental model.

Visualization of BubbleSort algorithm on a sample data.

So, to understand an algorithm, we should see it running on a sample data step by step, and then we should select the best data to not fall in a trivial example and this is another problem...

But, why we don't add a step in between ?
We have the algorithm coded in some language in one side, and the algorithm running in the other side.
Maybe adding the visualization of the algorithm itself can help the reader constructing quickly his mental model and reducing the time needed to see the algorithm running on data.

Some people tried to visualize algorithms by translating loops directly using schematics like these :

Explaining the multiplication µAlgorithm inside the µProcessor and introducing the Booth Algorithm.

Another almost useless technique for visualization I have seen in a site

But can we mix the algorithm itself and the data manipulation in the same place ? I have produced this figure from my own thoughts but really, we should think in a new way than everybody is following.

Searching for a new and better figures to help construct quickly the mental model.

In the last figure, arrows explain movement, the limits are nicely seen and understood, and it is clear that the second variable start its position from i and not from zero.

Can we ameliorate this figure and prove this way of representation scientifically ?

Maybe it can be a good point to start from.

Fitts' Law, a small presentation

2009-12-26T09:43:00.001+01:00

One of the most known and applied laws of Human-Computer Interaction is the Fitts' law. A good interface should optimize and reduce the time presented by this formula :

"a" and "b" are only two constants, defined by the used hardware properties like a mouse. D is the distance from the object and W is the object width.
I want only to say that this is only the first formula, many other derviate ones exist.

The most important is how to design interfaces to optimize the time to select an object. I want to present the openSUSE menu where the developers have applied this :

When I've clicked in Developpement option, all the menu has changed.

Being different from a "standard" menu, the user haven't to move his mouse by width, but only up and down and no new menu windows are created. So less mouse movement.

In this precise example, not only the Fitts' law is applied, but also the Steering Law (Accot-Zhai).

This is just an introduction, and presentation for a small example, but I will to show interface features in other projects.

Linux Graphics, Part one

2009-11-14T04:23:00.001+01:00

In this small post, I will speak about the Linux Graphic Stack and changes being addressed to it.

It is very clear, that the most thing that should change in Unix systems are the graphics. Most Unix systems use Xorg to handle all their graphics. I really think that is a very bad and old solution that should be changed.

The Past

To make things looking clear, lets compare the architecture of the current implementations :

In almost any Unix-like system, if we use Xorg it will be the layer responsible from low level graphics to some more advanced features. So it is doing a job which only the kernel should be responsible for.
Even graphics card drivers, they are made for every Xorg release and not for a Linux release.

If we add to this that X releases are not as frequent as the two other layers, we can conclude that the Linux graphics mechanism is the Achilles' heel of all the system.

Future

The future may be a lot better. With the include of the Kernel Mode Setting [1] in last Linux kernels (from 2.6.29) and specially in the 2.6.31, the kernel start getting his real role of the only thing accessing low level hardware.
The KMS allow less flicker, quicker terminal switching (with Ctrl-Alt-F1 to F7) and more.
Another feature is the ability to run Xserver itself as a normal application without root privileges. And allowing more security.

This start to let the Kernel do his job, is only the start. The Linux kernel 2.6.33 will include more work done to provide more low level handling : a KMS page-flipping ioctl [2] to optimize more and more time, and simplify management for other systems.

The page-flipping is a necessary thing for other projects like the Wayland Server, which is a lightweight replacement of X (and not a rewrite of it).
Xorg has become very old, new models of graphics and inputs come and its becoming more and more complex to make X supports them all.
Multi-touch input is a main area where X can't provide a good solution for that. Even Multi-pointer-X still doesn't answer all questions.

Wayland can become a solution if we design it having all new today problems in mind. It should be lightweight and easy to develop. After that, libraries like Nokia Qt could be easily ported to it.
The thing that everybody should know is that when a project becomes very old, with code that can't be understood, and huge, then it should be rewritten from scratch.
KDE developers faced many critics when they started rewriting KDE to release KDE4. But in the end, we now after releasing a "clean" desktop environment, we start seeing differences.

The design of Wayland, shouldn't be addressed without proposing a model being a skeleton of all manners of view surfaces and abstraction behind that. A recent paper [3] collected most of problems and it need to be applied on a real world implementation like Wayland.

To be continued...

Links :

[1] http://en.wikipedia.org/wiki/Mode-setting

[2] http://www.phoronix.com/scan.php?page=news_item&px=NzY5OA

[3] http://portal.acm.org/citation.cfm?id=1629826.1629828&coll=Portal&dl=GUIDE&CFID=61492784&CFTOKEN=12278186

Google, THE monopole

2009-06-13T23:56:00.000+02:00

The main goal of Google is to hold the maximum of viewed adverts on the net, and gain huge amount of money. Till now, ads are only shown as small links with some text. And the context intelligently matches user interests.

But, will google in the future, only rely on such type of incoming ?
My answer is simply : NO !

First step after acquiring Youtube is to play video ads before launching mostly viewed sequences, also put some intelligent context matching text ads somewhere. There is dailymotion and he already doing that.

But if we rethink another service that only google holds its sources, the GoogleStreetView !
Google Street View is a very helpful service provided by Google for free and for everybody.
Google can simply use some algorithms to detect the ad places in a street view, the orientation the deformation, and replace these zones with Google related stuff.

Goole StreetView, the next...

As explained in the photo, it won't be a difficult thing to be coded by the Mountain View team, and it will be a "Google only" stuff difficult to be reproduced in resonable time by other competitors. And of course, it will be a colossal source of money.

I really wish you "Don't be evil" Google !

Nanotechnologies & Nanostructured Materials workshop

2008-03-16T10:32:00.001+01:00

From yesterday (15 March) and until 21~22 I will be in Hammamet.

I am proud to be member of the technical staff of a workshop that speaks about nanostructred materials and nanotechnology.

This is the first time in Tunisia and it is with collaboration with many US universities.

I will attach photos later ! Please wait !!! ;-)

Journée Texas Instruments

2008-02-13T19:42:00.001+01:00

J'ai pas raté bien sûr le séminaire organisé par Texas Instruments le 7 Février à la Faculté des Sciences de Tunis

Voici une image du public présent !
(A ne pas oublier le grand nombre de la communauté open source qui a été présente)

Me voici avec mon Classmate Zied

Encore une fois

Et bien sûr nous n'avons pas oublié de prendre des photos avec Mathias le représentant de TI dans cette journée ainsi que mon ami Nizar !

SFD 2007 !

2007-10-30T14:52:00.000+01:00

Software Freedom day 2007 !
It was a very exciting event ! I discovered many things and I have met many persons that I had known only by mail.

I got many photos and I am proud to show some in my blog !
This is a photo with Mr. Hatem Zidi one of the best IT engineers in Tunisia !

Other photos are coming soon ...

Novell Hack Week

2007-07-20T12:30:00.000+02:00

J'ai entendu parler ces jours du Novell Hack week, c'est un événement qui a lancé Novell fin juin pour que ses ingénieurs et développeurs font ce qu'ils veulent faire !

Vraiment c'est très magnifique d'être payé pour faire ce qu'on veux ! et pendant toute une semaine ! De ce fait les développeurs ont pris un bon souffle et ont contribué à plusieurs projets.

Le résultat était en totalité bon mais il y a quand même des projets qui ont échoués !
En tout cas même pour ceux qui rien n'a marché pour eux, ils ont un prix de "Best Failure"

je ne pense pas avoir ce genre de chose dans les sociétés tunisiennes !

Link

Je google, Tu google, Il goog...

2007-07-07T16:00:00.000+02:00

Je pose parfois quelques questions sur mon utilisation de l'Internet. Et je trouve parfois des résultats magnifiques !

J'utilise dans 99% des cas le moteur de rechercher Google pour trouver quelque chose du Net,
Mon mail estsur Gmail,
j'utilise ce blog qui est un service de la société de Google,
Je suis inscrit à beaucoup de groupes hébergés par GoogleGroups,
Parfois je lis quelques pages des livres de la base Google Books,
J'utilise GoogleMaps ou GoogleEarth pour explorer les villes que je voulais les visiter,
J'utilise GoogleTalk pour le chat,
J'ai essayé GoogleDesktopSearch et il m'a plus beaucoup contrairement à celui de M$ qui m'a généré une base de donnée des références des fichiers plus grande qu'eux mêmes réunis,
J'ai utilisé GoogleAgenda pour une fois quand j'ai voulu voir le calendrier et que le PC que j'utilise ne m'a pas permi de faire ça (question de droit sous WinNT),
j'ai utilisé GoogleDocs pour ouvrir un document .odf dans un PC ne contenant pas OpenOffice,
j'utilise parfois GoogleTrends pour les statistiques,
J'utilise beaucoup YouTube ou GoogleVideo pour regarder les vidéos
Je pense que j'ai utilisé Google pour la traduction d'une page en autre langue
....

Je suis donc très relié à Google et très dépondant ! Et je pense que je ne suis pas le seul à être comme ça ! Mais à long terme Google va devenir le seul monopôle au monde, il va avoir tous les documents des utilisateurs entre ses mains que ce soit dans Gmail ou dans les documents des utilisateurs de GoogleDocument (tableur, éditeur de texte...),
Il aura aussi tous les statistiques de recherches,
Tous les puublicités,
Tous les livres numérisés,
Des Disques durs en totalité sauvegardés dans ses serveurs (un nouveau services encore Alpha...),
En deux mots : il aura Tous ! (ce sont 3 mots ?)

Ce jour la Google va il rester tranquille ? Ne va-il pas changer ses objectifs ? Ne va-il devenir méchant ??

Hey Google ! Don't be Evil !