First impressions of GNOME usability testing

I am delighted to have completed usability testing on 10 participants!

Generally speaking, the testing process went really well. There are of course some downsides to it. I’ll go ahead and share an unordered list of some things that went right and some that went wrong:

  • Covering a wide range of users

I’ve had people of different professions, age and computer expertise taking the test. This led me to really interesting and surprising findings which I will share in a future post, explaining in details using heat maps and charts. It was really interesting seeing different people taking different paths to accomplish the same tasks.

  • The right number of testers

Looking back at one of my previous posts where I talked about “Deciding the number of testers”  I explained a chart which assumes that 5+ testers are enough to uncover most of usability issues. I found this to be correct, during the testing sessions. As the number of participants kept growing I encountered less and less surprising results. Their difficulties on certain tasks became almost “predictable”. So 5+ testers it’s really enough, if you don’t want to get into details.

  • The right number of tasks

Just before starting the tests I regretted the fact I had only written 16 tasks, thinking that I could include more, therefore cover more features. After the first test I changed my mind. Considering the fact that I spent some time before the test talking about GNOME and explaining the testing process, also answering some questions and including the follow-up questions, 16 tasks were just enough. It would be very hard to keep the participants concentrated for a longer period of time.

  • Blame it on usability

Even though in my pre-test script I explained to each participant that the tests purpose it is not testing their knowledge but it is testing GNOME’s usability instead, I could see they still felt uncomfortable when they weren’t able to accomplish a certain task immediately. So I had to repeat again that it is okay to not be able to accomplish the task, just blame it on usability 😉

  • Thinking out loud and recording

Another point where participants felt uncomfortable was communicating their thoughts. Most of them did it only during the first tasks and then stopped. So I sometimes had to ask questions like “Could you please tell me what are you thinking this moment?”. Other times when I did not want to interrupt the participants, I asked them in the follow-up questions like”What were you thinking during this task?”, “Why did you make that choice?”…

Participants also felt uncomfortable to record their faces or voices during the test, so I only recorded the screen and wrote some notes during each test, so I can refer to them during the analysis.

  • Struggling with the tasks

I am really happy with the fact that the participants found the tasks clear and straightforward and accomplished most of them successfully, but there were some tasks that they all struggled to accomplish. Even though I mentioned in the beginning of the test that I can not help with the tasks, I am here just to observe. Yet they constantly asked for help. I noticed that this is a challenge for others who do usability testing too. In this case I was not an exception. It was really hard not to give hints, specially when they were so close to finish the task.

  • Yes to GNOME!

Except the tasks every participant had to accomplish on Photos and Calendar, I also let them explore GNOME a little bit. For example I let them relaunch the applications when they accidentally closed them, let them search for files(of course I offered help in case they needed, since this Is not part of the test). In the end I asked them about their overall impression of GNOME and if they would use it daily, and they all answered positively. They also asked a lot of open source related questions and some where interested to start contributing to open source projects.

 

So this was just a short first impression of the usability testing I conducted. There is a lot more to cover. I am planning to write more posts, that way I can get into details of the test.

See you soon!

By:

Renata Gegaj

Posted in:

4 responses to “First impressions of GNOME usability testing”

  3. These are great first impressions! I love that you found “5+ testers” to be correct, and that you were able to “predict” results in later tests. I experienced this in my tests, as well. After about five testers, you usually have “good enough” data that you can make improvements. And of course, it’s iterative: design, usability test, tweak design, another usability test, etc.

    I think the “5+ testers” rule is very significant in usability testing. You don’t have to put a lot of effort into usability testing to get enough data about how to tweak your design. Anyone can do this!

    I also liked your comment about the right number of tasks. You were worried that 16 scenario tasks wouldn’t be enough tasks to keep your testers occupied for the whole time, that they might be able to do them very quickly. Of course, you assumed this because you have been working with GNOME as part of your usability work, so you know these programs very well. And this is also why developers are often not able to guess how easy their software is for other people to use; the developer *wrote* the program, so knows how the menus are laid out and how to access all the functionality. A new user does not, and may need to discover the functionality.

    That’s why usability testing is so important, especially in open source software! We cannot always predict how easily other people can use our software, so we need to do usability tests on our designs so that we get it right. And we need to keep doing them, iteratively, to make our designs even better.

    Great job! I’m looking forward to your analysis! A reminder: I find it’s best to do the analysis in two parts: a heat map, and a discussion of themes. To create your heat map, use the method I describe on my blog about how to create a heat map. (Search for “heat map” if you don’t have the link.)

    You may already have an idea of themes: What did testers find easy to do? What did testers find to be more difficult? Starting with the heat map, themes may become obvious. Look for “hot” rows (lots of black, red or orange). “Cool” rows with lots of green or yellow are likely to be areas with good usability. Pull quotes from users during the tests to give examples of problems in “hot” rows. What was common between these “hot” rows? Were users confused by the same or similar design patterns in these difficult areas? Also look at what went well: What was common between the “cool” rows? Why did users find these easy to do?

    Liked by 1 person

    Reply

Leave a comment