Diana Lipcanu

Interaction Museum

Both Interface Designers and Human-Computer Interaction (HCI) Researchers build interaction techniques. The main difference is that researchers experiment with new ways of interaction, while UI designers look for the most user friendly patterns.

The Interaction Museum is an inspiration blog, where interaction techniques from the HCI research literature are presented in an accessible way to designers. The purpose is to create a place in which both professionals can unite forces to build better interaction.

Project Type

Master thesis & Website

Total Time

6 months

My role

User Research, Design,
Front-End

Worked with

Isha van Baar

Context

The Interaction Museum is the brainchild of Check out Wendy Mackay's publications here Wendy Mackay and Check out Michel-Beaudouin Lafon's publications here Michel-Beaudouin Lafon . In 2006 they wanted to build an extensive collection of An interaction technique is “the fusion of input and output, consisting of all software and hardware elements, that provides a way for the user to accomplish a task.” interaction techniques from the HCI research literature. Unfortunately they could not complete and publish the project, despite interest from the research community, as the web was not yet ready for such a media-heavy resource. Ten years later, under the supervision of Wendy & Michel, Isha van Baar and I took on the challenge to design and develop a new, improved Interaction Museum as part of our Master Thesis Internship, within More about Inria, here Inria and Universite Paris Saclay.

HCI Researchers & Interaction Techniques

In HCI researchers develop and evaluate interaction techniques to push the boundaries of computer interaction. They then publish them in research papers. Few times, other researchers build upon these methods. Yet it rarely happens for designers outside research to pick them up and implement them in commercial products.

Conformism in Commercial Interfaces

On the other hand, most commercial interfaces (desktop, web & mobile) have largely stayed the same for the past three decades. Windows, Icons, Menus and Pointers (WIMP) with basic Direct manipulation is an interaction style which involves continuous representation of objects of interest direct manipulation functionality (‘drag-and-drop’) are the standard. The main reason for this, is that a good interface for large audiences is one that is easy to understand and Nielsen’s usability heuristics have set the standard for good design for more than 20 years familiar to novices .

The ideal in commercial software is that users only learn a small set of interactions that become standards that any designer can reuse. To support this, designers follow usability guidelines advocated by leading software platforms such as Check out Apple's design guidelines Apple , Check out Google's material design Google , and Check out Microsoft's material design Microsoft . Interface programmers also prefer implementing ready-made interaction techniques from UI Kits (JQuery UI, Check out JavaSwing Java Swing , etc.).

The standard jQuery Widgets, that are used extensively throughout the web

Yet, these standards and guidelines are not always the right solution. Buttons, drop-down menus, switch buttons and drag-and-drop are not always enough to support complex or unusual user tasks. In some cases, it is more appropriate to use custom designed interaction techniques for a specific task or user group.

Design Brief

Wendy & Michel noticed that there is a great pool of unusual, creative interaction techniques in the research field, and a lack of interaction diversity in commercial interfaces. Thus, they wanted to bring the two worlds together, with an online collection of non-standard interaction techniques, called the Interaction Museum.

The Interaction Museum is also envisioned as an active community that maintains and updates this collection, and a content management system that allows members to donate their work to the collection. The scenario bellow visualises how the Interaction Museum was meant to work.

Created by Simon Child, from the Noun Project

Rose

The HCI Researcher

Rose is fascinated with interaction techniques and has invented a new smartphone gesture called ThumbFlow. She published it in a paper at the UIST Conference.

A day in the life...

Rose used the Interaction Museum when she wrote her paper on ThumbFlow to help her find similar interaction techniques for her literature review. She thinks ThumbFlow is a great invention, as she proved it to be two times faster than standard selection methods for big data sets. She even thought of several use cases as part of common smartphone apps. So now Rose adds her interaction to the Museum, so others can pick it up for their literature reviews, or use it in their products.

Created by Simon Child, from the Noun Project

Dan

Interaction Designer

Dan is designing a new calendar app. His requirement is to build a date picker that will differentiate his product from all other calendars on the market.

A day in the life...

Dan has been looking at all date pickers he could find. But nothing caught his attention: there's not much variation. He remembers tagging a blog post about a collection of unusual interaction techniques from research literature. He finds the Interaction Museum and stats browsing it. An Exhibition called Selection Methods on Touch Interfaces attracts his attention. There he finds ThumbFlow, which inspires him to build an amazing date picker. His project manager is impressed and users are delighted.

Imagining an Ecosystem

When Isha and I took on this project, we had the freedom to reimagine it. We feared the old concept was out-of-date, so the best place to look for answers was with the users. For that we first had to define who our users might be, and how would they interact with each other.

Our first assumptions about a potential user ecosystem

First Interviews: Debunked Assumptions

To find out if our assumptions about the users were true, we went directly to the source, and we recruited:

  • 4 designers
  • 3 HCI Researchers
  • 3 HCI Professors
  • 3 HCI Students

...and we interviewed them about their experience with interaction techniques. After we got the data, we analysed it through an ad-hoc version of imported from social sciences, grounded theory is useful for analysing qualitative data from user interviews. Read more about how it works here. grounded theory analysis grounded theory. As we were still figuring out which are the right questions to ask, and we also had only 3-4 participants of each user group we considered these pilot interviews.

Participant's answers visually calssified into categories.

The most interesting findings were:

Disclaimer: these findings apply only to our interviewees, they cannot be generalized.

  • Researchers are motivated by the recognition of their work within research, but they do not necessarily want their work to be used in commercial products
  • Researchers don’t like to put extra effort in a published project, to communicate it to others outside research (like adding it to the Interaction Museum)
  • Students did not find inherent value in helping the building a collection of interaction techniques
  • Designers base their choices on what users already know and they don’t want to design something that developers can’t program
  • Designers do not read research articles but look for inspiration in clients’ style, design social media and competitors’ interfaces
  • Interaction techniques are hard to communicate and teach, without the use of video or working prototypes

Researching Researchers

Interaction techniques from HCI research are the building bricks of the Interaction Museum. Thus we had to find out more about how researchers create, use and distribute them. For that, we sent out a questionnaire to several research labs and lead researchers.

We got 43 complete responses. 95.3% of respondents were affiliated with a University, and 57% had more than 8 years of experience in the domain.

Findings: Invention & Development

The most common sources of inspiration for interaction techniques

  • Observing users (44.5%)
  • Exploiting a technology feature (44.5%)
  • Improving an existing technique (35.8%)

Researchers mainly develop techniques for:

  • Desktop computers (33.3%)
  • Custom devices (27.4%)

...but also for tablets (15,6%), smart phones (13,7), tabletops and smart watches.

Only four interaction techniques (out of 38) were developed with web technologies. Java is the most used programming language (31.6%). This denotes a trend in the field, probably due to Java being a common language taught in academia.

Findings: Sharing Interaction Techniques

The most widespread medium to share interaction techniques is research publications (78.9%). Even if interaction techniques are best communicated by dynamic media, only 47.4% of respondents shared videos, and merely 31.6% had interactive demos. We expected that at least few researchers would have had their interaction technique implemented in a commercial product, but none did.

I was satisfied with the response from sharing the interaction technique as a ...

‘Conference presentation’, ‘Video’, and ‘Interactive demo’ outperform all others in the satisfaction to dissatisfaction ratio.

I would like others to use my interaction technique

Researchers generally want others to build on their work, but only 3 got a response from commercial developers.

Findings: Finding Interaction Techniques

Most respondents indicated that they look for interaction techniques occasionally, but intensively. This fits with the task of doing a literature review for a research paper.

Finding techniques that they don’t know is a difficult task

Implications for Design

The Interaction Museum should cater to the need of finding interaction techniques easier. For that we need a wide range of techniques linked together by similarity.

Choosing the right media to exhibit interaction techniques in the Interaction Museum has been a great challenge. Through this study, we wanted to find out what media do researchers create for their work, to assess our options. Their answers confirmed our difficulties in finding videos and working prototypes.

Yet, the most disappointing was that very few participants worked with web-based technologies. As the Interaction Museum is web-based, and Java no longer works on major browsers, this leaves us with little hope of exhibiting many working interaction techniques.

How do Interaction Designers Design

We wanted to understand why so many designers choose standard interaction in commercial software. For that we interviewed six designers who work with interactive interfaces. We we used the critical object interview method, pioneered by Wendy Mackay asked them about recent projects they worked on, that contained interaction techniques.

Methodology

For each interaction technique, I arranged the interview data chronologically in an Similar to Jalal and colleagues’ ‘color portraits’ ‘interaction story’ , containing quotes about the design process and pictures. In total, we gathered examples from 13 projects, in which I identified 20 interaction stories. I then used the grounded theory approach to analyse the data.

To illustrate how categories apply to 'interaction stories' and to get an overview, I conceived a visualisation model.

Story Visualization

story visualization

Interaction Story

Check out this amazingly creative design studio from Paris Studio Fables participated in a competition to design the French Ministry of Culture’s 2016 digital brochure. Their concept implied a long picture that the users could scroll horizontally.

The design started with a user constraint (user constraint) - the interaction had to be discoverable for non-tech savvy users. This triggered a design constraint (design constraint) - the best solution would avoid “pressing on the phone”. So their solution was using the gyroscope to detect any tilt of the phone, which would trigger the long image to be scrolled (non-standard solution). I categorised this solution as non-standard.

When implementing the technique, they identified another design constraint (design constraint):

you don’t see in front of you the picture (when you tilt the phone), and if there’s light outside, you can’t even see the screen

To solve this issue they got another idea, inspired from a metaphor (non-standard solution):

[we wanted to have] the user in the middle of culture, so you move around yourself to discover all the people working in culture

Yet, once they implemented this solution they encountered two technological constraints: it was not working on older devices (technical constraint) and it kept breaking down (technical constraint):

[It was] complicated [to implement] because we are not developers

Due to these constraints, they finally decided to go with a standard image slider, which the user had to interact with by swiping (standard). The designer wasn't satisfied (standard) with the final solution:

in terms of interaction, it was not the most interesting

Category: Design Solutions

All our interviewees had a highly iterative design process. Before a final version, designers have prototyped and discarded many versions and alternatives. I looked at how these solutions evolved, what influenced them and why did some solutions get discarded.

Design Solutions

Category Dimensions

Standard

a solution that is by default implemented in a majority of similar systems

Non-standard

a solution is that is less common or completely unique

Released

a solution adopted in the final product

Discarded

a solution abandoned along the design process

Satisfied

the designer is satisfied with the solution

Neutral

the designer expressed no particular feelings towards the solution

Dissapointed

the designer was dissapointed with the solution

We deliberately recruited very creative designers, so it came as no surprise that 4/6 explored non-standard interaction techniques. Actually, we gathered 16 (out of 20) stories in which designers explored at least one non-standard solution.

Almost 2/3 of non-standard alternatives end up discarded, as opposed to one-half of the standard ones.

Out of the 20 interaction stories in:

  • 13 designers worked with

    desktop computers & mobile phones

    common devices
  • 7 designers worked with

    a gestures triggered harp, a wall-sized display, a custom device with nine buttons, and a custom tablet built before the iPad

    uncommon devices

Surprisingly, when designing for common devices, designers experimented with non-standard interaction techniques almost as much as with standard ones. Yet, non-standard techniques didn't necessarily make it in the released version.

Designers were more satisfied with non-standard solutions they designed, than with standard ones. An explanation could be the Read more about the IKEA effect IKEA effect cognitive bias : people value artefacts they build themselves more than products that are ready-made.

Constraits

In this study, a constraint is a challenge or limitation the designers faced.

Category: Constraints

Category Dimensions

Pivotal

it triggered a new design alternative

Inconsequential

it didn’t affect the design solution

Type

Design

considerations on how the user experiences the product e.g. discoverability

Technical

breakdowns that depend on limitations of software, hardware, and programming skills

User-related

characteristics of target users that shape design considerations

Business

decisions taken by clients or project managers

Time

limitations imposed by deadlines

Category: User Feedback

The designers we interviewed were very sensitive to user feedback. In four cases, negative user feedback led to new design constraints, which triggered their design to change from a non-standard solution to a standard one.

When do designers design non-standard solutions?

Most times, desigers built non-standard solutions because of the uncommon technology they worked with. The second most influencing factor was the desire to delight the user. As one participant noted:

There’s always a more traditional, and not delightful way to do something, but it’s better if we’re bringing something new to the user

When are standard solutions chosen over non-standard ones?

There were three interaction stories in which designers started by building a non-standard solution, but in the end they had to re-design a standard alternative for the final product

The most common reason why a non-standard interaction technique wasn't implemented was because of a technical constraint (4 cases). Implementing a non-standard technique is much harder than a standard solution, that can be found in a UI Kit. One designer said:

Using something that's built-in, it's a 5-second work, build something, even if it's a simple button, it is about 5 days of work, if you build it on your own, from scratch

Other common reasons for abandoning non-standard solutions are:

  • User Feedback,
  • User Constraints
  • Discoverability

These three go hand in hand: designers tested their non-standard technique with users, and found a new user constraint, so they had to change their design accordingly. As one interviewee said:

Users don't want to make the effort to discover a new experience every time they try something new. So they are very happy with sticking to what they know already

Correcting Assumptions

Initially, we assumed that commercial software doesn’t have many custom built interaction techniques because designers don't have access to inspiring examples. Yet none of the stories of this study indicated that. On the contrary, designers did try out creative solutions, but user feedback and design, technical and user constraints persuaded them to choose standard solutions.

As we did this study towards the end of the internship, we did not have the time to implement the necessary changes in the Interaction Museum

Implications for Design

To support interaction designers in creating non-standard interaction techniques, the Interaction Museum should provide resources to help with the technical implementation of the technique.

Designers are also attracted by delightful interaction. Thus, to cater to them, the Interaction Museum needs to present its content in a visually pleasing way.

Generative Design Cycle

During this project, Isha and I have gone through the Generative Design Cycle five times. Each cycle we gathered user feedback, re-defined the design space, improved previous prototypes, and evaluated the system.

The Generative Design Cycle is a visualization of the design process proposed by Wendy Mackay

Paper Prototyping

After gathering initial user insights, we defined our design space and prototyped on paper. We started with paper prototypes because they are the easiest to change and iterate on. We didn’t want to get stuck on one idea before we explored alternatives.

Video Prototype on how the Interaction Museum would work.

Video Prototype on how adding a new technique would work

Web Prototype

The next four iterations have been directly on the web. We used Kirby is a file based CMS Kirby as a Content Management System.

To check out the final working website, click here

Design Challenges

1. The basic unit

One of the first challenges we faced was deciding what the smallest unit of the Museum is - the interaction technique. There is no agreed upon taxonomy of interaction techniques, nor an encompassing definition.

Is multi-touch interaction the smallest unit, or is it swiping? Should swiping left be a different entry from swiping right? Having a strict rule of what the basic unit is would have made it difficult for contributors. So we decided to let them choose what they believe the basic unit is.

2. Ain’t nobody got time for that

One of the biggest issues with the initial, 2006 Interaction Museum, was the amount of required media for one technique. Many details were mandatory to have a comprehending understanding of how a technique works:

  • Long & short descriptions
  • Storyboards
  • Videos
  • Images
  • Technical descriptions
  • Working prototype
  • Related Literature
  • Related Interaction Techniques

But as we found out in our study, researchers rarely create all these artefacts when building an interaction technique. Also, after they publish a paper, researchers don’t want to take the time to complete such a long form.

So we decided to just require few elements, such as a title, an image, a short description and tags. The rest were optional, for those who had the time. Even so, adding a comprehesive entry still took a lot of time.

3. Visualising interaction

As interaction is happening in time, visualising it only with still images was insufficient. For that we turned to GIFs - short repeating animations that are perfect to represent at a glance what an interaction technique does.

4. The issue with taxonomies

Once we had a bunch of interaction techniques in the Museum, we needed some sort of categorisation. Yet, we didn’t want to impose a taxonomy, as that could divide and confuse our researcher user base. Moreover, there’s no taxonomy that could accommodate every existing interaction technique.

Thus we implemented 'collections': a way to gather any group of interaction techniques, as the user sees fit. This way, we hoped that eventually a natural taxonomy will emerge.

Evaluation

Even if we’ve done smaller evaluations along the design process, the most interesting one was showing the museum to interaction designers. We didn’t particularly want to conduct a usability test, but more of a concept evaluation.

Out of our five interviewees:

  • One designer expressed a desire to use the Interaction Museum in his work.
  • Other three thought it was too ‘far away’ from their

    They worked only on desktop and mobile. At that time, the museum had a lot of gestural and VR techniques

    usual projects
  • Three thought it had potential, but was not ‘there yet’

We also found several usability issues, but one of the most striking was the lack of a good way to start exploring the museum. The most requested feature was a filter functionality.

This is my proposed design for a filter function. We didn't have time to implement it.

The feedback we got for the Interaction Museum was disappointing. The issue with mainstream technology, such as desktops and mobiles, is that strong standards rule. Changing those standards would require a paradigm shift, and no collection of inspirational examples can do that on its own.

Even if interaction designers seemed like the natural target audience, we should have focused even further, on designers working on Such as VR, AR, IoT, smart robots. Click more to see Gartner's 2016 Hype Cycle new technology that is in the hype cyle spotlight . They, together with researchers, can create the community the Interaction Museum needs to thrive.

Retrospective & Lessons Learned

In the end, the Interaction Museum wasn’t officially released by the time Isha and I finished the internship. The web app was buggy, and we didn’t manage to collect enough interaction techniques. Michel and Wendy undertook again the project, and looked forward to assigning new interns to push it further.

Nonetheless, what we left behind was

  • a working app,
  • a clear concept,
  • user studies that future decisions can be based upon,
  • a lot of documentation,
  • and two satisfied supervisors.

I learned how to conduct proper user studies, to analyse the data, present it and to build a theory on top of it. I also learned how to deal with projects in their infancy: taking them from mere concepts to actual realities.

But what affected me the most, in my practice as a designer, was being exposed to all these creative and uncommon ways of designing human computer interaction. Now I know that whenever I will sit down to design a new product, I will look beyond design standards and UI Kits, to better ways, shaped by target users and their tasks.

Check out the Tabletop Match case study to read more about my first user studies.