I am delighted to have completed usability testing on 10 participants!
Generally speaking, the testing process went really well. There are of course some downsides to it. I’ll go ahead and share an unordered list of some things that went right and some that went wrong:
- Covering a wide range of users
I’ve had people of different professions, age and computer expertise taking the test. This led me to really interesting and surprising findings which I will share in a future post, explaining in details using heat maps and charts. It was really interesting seeing different people taking different paths to accomplish the same tasks.
- The right number of testers
Looking back at one of my previous posts where I talked about “Deciding the number of testers” I explained a chart which assumes that 5+ testers are enough to uncover most of usability issues. I found this to be correct, during the testing sessions. As the number of participants kept growing I encountered less and less surprising results. Their difficulties on certain tasks became almost “predictable”. So 5+ testers it’s really enough, if you don’t want to get into details.
- The right number of tasks
Just before starting the tests I regretted the fact I had only written 16 tasks, thinking that I could include more, therefore cover more features. After the first test I changed my mind. Considering the fact that I spent some time before the test talking about GNOME and explaining the testing process, also answering some questions and including the follow-up questions, 16 tasks were just enough. It would be very hard to keep the participants concentrated for a longer period of time.
- Blame it on usability
Even though in my pre-test script I explained to each participant that the tests purpose it is not testing their knowledge but it is testing GNOME’s usability instead, I could see they still felt uncomfortable when they weren’t able to accomplish a certain task immediately. So I had to repeat again that it is okay to not be able to accomplish the task, just blame it on usability 😉
- Thinking out loud and recording
Another point where participants felt uncomfortable was communicating their thoughts. Most of them did it only during the first tasks and then stopped. So I sometimes had to ask questions like “Could you please tell me what are you thinking this moment?”. Other times when I did not want to interrupt the participants, I asked them in the follow-up questions like”What were you thinking during this task?”, “Why did you make that choice?”…
Participants also felt uncomfortable to record their faces or voices during the test, so I only recorded the screen and wrote some notes during each test, so I can refer to them during the analysis.
- Struggling with the tasks
I am really happy with the fact that the participants found the tasks clear and straightforward and accomplished most of them successfully, but there were some tasks that they all struggled to accomplish. Even though I mentioned in the beginning of the test that I can not help with the tasks, I am here just to observe. Yet they constantly asked for help. I noticed that this is a challenge for others who do usability testing too. In this case I was not an exception. It was really hard not to give hints, specially when they were so close to finish the task.
- Yes to GNOME!
Except the tasks every participant had to accomplish on Photos and Calendar, I also let them explore GNOME a little bit. For example I let them relaunch the applications when they accidentally closed them, let them search for files(of course I offered help in case they needed, since this Is not part of the test). In the end I asked them about their overall impression of GNOME and if they would use it daily, and they all answered positively. They also asked a lot of open source related questions and some where interested to start contributing to open source projects.
So this was just a short first impression of the usability testing I conducted. There is a lot more to cover. I am planning to write more posts, that way I can get into details of the test.
See you soon!