GNOME Usability Test Results (Part 2)

gnome_carbone_by_vicing

Here I am with the last part of analysis🙂
On part 1 I focused more on visualizing the results, demographics and talked more about the methodology I used for testing. Later we took a closer look on how testers performed on every task given, using a heat map.

Heatmap

This time I am coming with more details “pulled” from the heat map. I find it is best if I break down the analysis into two sections: ” What went well?” And” What were the challenges?”. In these sections I’ll include the tasks that performed well and the tasks that were more challenging and give a brief description for each of them.

In the “What went well?” part I am going to analyze the “cool” rows. Basically the green and yellow boxes, which represent the “easy” and the “somewhat difficult” tasks.

In the “What were the challenges?” section I am going to talk about the “hot” rows. That includes the orange, red and black boxes. These colors represent the tasks that were “difficult” to accomplish.

 

What went well?

Photos:

Edit a picture(task 4): After showing your friends all the pictures of the “Thailand trip” album, you revisit your “favorite” picture since you want to edit it before sharing on social media.

This task turned out to be straightforward and intuitive for all the participants. They said that the icon used for editing associated with its function and it is also familiar to them from other photo editing applications that use similar icon.Screenshot from 2016-08-24 20-22-55

Crop the picture(task 5): You start by cropping the picture to make it look smaller.

We can notice some yellow boxes in this task and that is because of the confusion caused by the sidebar  “aspect ratio” options for cropping the picture. Participants thought that they can adjust the size of the picture only using the ratios in the sidebar and seemed to be confused. After that, they tried clicking the picture and adjusting from there.Screenshot from 2016-08-24 20-42-56

Change the colors(task 6): Than you want to play with colors a little bit. You want to make the picture brighter and lower its saturation but leave the contrast as it is.

Enhance the picture(task 7): Now you want to enhance your photo. You sharpen it a little bit and leave the denoise as it is.

Apply filter (task 8): And finally, you choose a filter named “Calistoga” as a final touch.

All participants considered the editing process very simple. All testers managed to complete these three tasks without any difficulties. They were really easy to follow once they’ve found the editing icon.

Apply changes(task 9): You want to apply these changes, so the photo remains as it is.

I think it would be a better idea if I merged this task with of the previous task because most of the testers applied the changes before, so this task was not applicable in most cases.

Regarding the applying the changes process, some of the participants said that they expected an “Apply” or “Save changes” button instead of “Done”. They also expected to find the “Done” button in the bottom of the screen.

Done

Calendar:

Set an alarm(task 3): Set a reminder 10 minutes before the meeting starts, just in case you forget that you have a meeting.

After finding a way to create the new event, this task was really easy to accomplish. No participant encountered difficulties on this task.

Search for an event(task 5): 5.You have already created an event named “GUADEC” for the GUADEC  conference you are attending next month but you forgot the exact date. Can you please search for it and then tell me the date.

This task was also intuitive for all the participants, they found the search icon to be very familiar. Screenshot from 2016-08-26 01-19-40

What were the challenges?

Photos:

Create an album(task 1): You just got back from your trip to Thailand. You want to show your friends all the pictures that you captured there but you notice that they are all mixed with the other pictures that were previously there. To avoid the confusion you decide to create an album that contains only pictures from your trip and name it “Thailand trip”.

This task was definitely the most challenging to accomplish. All participants went through similar steps to complete this one.

First they clicked “Albums” on the view switcher and then tried clicking “Name your first album”, thinking that this option would create a new album.

AddAlbum.png

That didn’t work, so they tried right clicking on the screen which also didn’t work. After that, they looked for a “Create an Album” option in the “Application menu” and could not find it. Some participants even searched for “Create an Album” in the search bar.

Finally, they tried right clicking on “Photos” and found the “Add to Album” option on the bottom of the screen.

Another confusing thing for all testers was the “Organize” window. After they selected the desired pictures and clicked “Add to Album”, the “Organize” window would appear, giving two options: “OK” and “Add”. They immediately clicked “OK” and were not sure if they’ve created the album or not.

Organize

They went back to “Albums” on view switcher and found out that the album was not saved. So they started the process again, using the “Add” option this time. The participants also complained about not having a “Cancel” button on this window.

Favorite a picture(task 2): As you are showing the pictures of the album you just created, you notice a picture that you’d like to share on your social media accounts. You “favorite” that picture, in order to access it more easily later on.

This task was pretty easy to accomplish. The star icon was noticeable for nearly all testers. In some cases participants completed this task by using the selection mode. Although some insisted to right click on the picture and expected a menu where they could click “Favorite”.

Favorite-selection.png
Favorite a picture using selection mode

Favorite.png

Delete a picture(task 3): While looking trough all the photos, you notice that two of the photos look very similar, so you decide to delete one of them.

Set a background picture(task 10): You really like the way that the picture turned out and you decide to set it as a “Background picture”.

On both these tasks, most testers tried clicking on the picture first, then right clicked on it and expected a menu where they could find these options. It made more sense to them to have a special menu that they could use to operate with the picture instead of using the header bar menu.

dl.png

Calendar:

Add a new calendar(task 1): You want to have all your work related activities in your calendar but you don’t want them to be mixed with other activities, since you have a lot going on lately. In order to keep your events organized you decide to create a calendar only for work related activities. Name the calendar simply “Work” and make the activities appear in purple color.

In general, all participants were confused by the term “Add a new Calendar” or “Create a new Calendar”. They would expect for an option like “Add a new Calendar” to be located somewhere in the application menu or in a “more obvious” location and not under the “Calendar Settings”. Some even created a new event instead of adding new calendar. Almost all testers were not sure which option to choose for adding the calendar.

Calendar11

Testers expected feedback after they named the calendar. They complained about not having an “Apply” button since they were not quite sure if they added the new album.

 

Calendar

Create an event & set the time(task 2): You have a meeting with your boss today. You want to create an event named “Meeting for work”, the meeting starts at 2:00pm until 4:00pm. Make sure to put it in the “Work” calendar that you created earlier.

This task is derived from the first task, which was challenging to accomplish for most of participants. Therefore this task also caused a little confusion.

After adding a new calendar, testers thought that in order to create an event they need to somehow switch from the current view of calendar to the new calendar they’ve created, only then they’d have the chance to create a new event.

Two participants tried to use “Manage your calendars” to create the event. They tried leaving checked only the calendar they created and unchecking the other calendars, thinking that would “switch” the calendars.

Calendar3

One participant went back to “Calendar Settings” and clicked on the “Work” calendar he created earlier expecting a “Create an Event” window to appear.

Calendar4

After exploring a little, they found out few ways to perform this task.

Nearly all participants used the “quick add” popup to create the events, and found it very clarifying since they could pick the calendars they created and add the event there.

Calendar5

Just a few of them used the Application menu and the header bar menu to create the events.

Calendar6Calendar7.png

 

Setting a reminder was really easy for all the participants.

Look for a specific date(task 4): You plan to throw a big celebration party for your birthday next year. You want to check what day of the week will your birthday be, while wishing for Friday so everyone can show up! Check the calendar and tell me what day of the week your birthday will be in 2017.

For some reason, not many tester used the view switcher to skip to the next year. They used the arrows to change the current date and some tried to scroll through the months.Calendar 9Add an online account(task 6): You don’t want to move across different calendars, creating same events multiple times. So, you try to connect this calendar with google calendar (or any other online account you use).

The first thing some testers did to is choosing the “synchronize” option on header bar menu.

Calendar9After the testers found a way to add an online account and finished this task successfully, they wanted to remove the account.

Calendar10

The “+” and “-” signs were not that obvious to indicate their meanings. They would rather prefer something like “Add an Account” and “Remove an Account” buttons.

Few suggestions

After putting together these results I was thinking, what would I do to improve the usability of Calendar and Photos. So I thought I’d share a few suggestions🙂

Photos:

One thing I noticed while the participants started the “Create an Album” task is that as soon as they got into the selection mode the blue header bar grabbed their attention. They expected the “important” options like adding an album, printing or deleting to be located there, so they totally ignored the Action bar.

Photo1.png

I suggest adding a color to the Action bar, that way it would be noticeable enough.

Same problem with the “Organize” window. The “OK” blue button draws too much attention, so all testers would click “OK” instead of “Add”.

Organize
Calendar:

I use Calendar daily for quite some time now, so I can talk from a users perspective for a moment. I’d love to see the events highlighted instead of the dots underneath the numbers. That would give a clear look of the month.

Screenshot from 2016-08-26 15-39-59

 

This is it for my usability testing results and analysis, I hope it will help improve usability of Calendar and Photos!
Thanks for reading ^_^

GNOME Usability Test Results (Part 1)

gnome_logo_3d_by_ilnanny

This is the first part of analysis for the usability test I recently conducted, with the purpose to uncover usability flaws of two GNOME applications: Photos and Calendar.
For this part I am focusing on visualizing the results, demographics and talk more about the methodology I used for testing. We will take a closer look on how testers performed on every task given, using a heat map. Hopefully this will create a clear picture of the testing process and help to “get to know” the participants and understand them better!

Ten volunteers participated in the usability test, representing a mix of genders (slightly more men than women) and an age ranges (mostly at age 15 to 25).

Gender-f

Age

All participants were asked about computer usage frequency and the majority said that they use it on daily basis.

Usage

Participants identified their computer expertise by choosing a level between 2-5.

1-I don’t know anything, I never use the computer

2- I don’t know a lot, I am not a frequent user

3- I know some things

4- I am pretty average

5- I am better than most

Expertise

 

Most of the participants were not familiar with GNOME. Some of them had heard about it in the past but this was the first time using it, three of them use GNOME daily.GNOME Usage

Testers were provided a  laptop running GNOME 3.20 on Fedora 24 operating system, without any modifications that could possibly affect the participants overall experience during the test. They executed the test separately, using identical settings. I used Gnome Continuous image running on GNOME Boxes, to test with Calendar. Each participant used a separate guest  account that had been pre-loaded with the example files needed to complete the tasks.

Each usability test session started with an introduction:
Thank you for coming today!

Just like Windows is a desktop system and MacOS is a desktop system, GNOME is a free software desktop system. And you’ll be doing a few tasks using GNOME and a few GNOME applications. I’ll give you the tasks one at a time. We aren’t “testing” you. The test is all about the software. If you have problems with part of the test, that’s okay; that’s exactly what we are trying to find. So don’t feel embarrassed if you have problems doing something and please do not feel pressured by time or anything else. If you can’t figure something out, that is perfectly okay and will still provide us with useful information for the test.  If you have any questions about the tasks I will try to answer them, but the answers and feedback should come directly from you as much as possible so I will avoid anything that would lead you to a specific choice.  Also, I’m going to take notes while you’re doing these tasks. It will help me if you talk out loud when you are doing something, so I can take notes. So if you’re looking for a Print button, just say “I’m looking for a Print button.” And move the mouse to wherever you are looking on the screen, so I can see where you’re looking. It would also be very helpful for me to do a screen-record of the testing process, so that I can go back to that later during my analysis.

 

After that, participants were presented with 16 scenario tasks which they were expected to complete in 40-50 minutes. All the participants managed to complete the tasks in less than 50 minutes.

Photos:

1. You just got back from your trip to Thailand. You want to show your friends all the pictures that you captured there but you notice that they are all mixed with the other pictures that were previously there. To avoid the confusion you decide to create an album that contains only pictures from your trip and name it “Thailand trip”.

2. As you are showing the pictures of the album you just created, you notice a picture that you’d like to share on your social media accounts. You “favorite” that picture, in order to access it more easily later on.

3.While looking trough all the photos, you notice that two of the photos look very similar, so you decide to delete one of them.

4.After showing your friends all the pictures of the “Thailand trip” album, you revisit your “favorite” picture since you want to edit it before sharing on social media.

5.You start by cropping the picture to make it look smaller.
6.Than you want to play with colors a little bit. You want to make the picture brighter and lower its saturation but leave the contrast as it is.

7.Now you want to enhance your photo. You sharpen it a little bit and leave the denoise as it is.

8.And finally, you choose a filter named “Calistoga” as a final touch.

9.You want to apply these changes, so the photo remains as it is.

10.You really like the way that the picture turned out and you decide to set it as a “Background picture”.

 

Calendar:

1.You want to have all your work related activities in your calendar but you don’t want them to be mixed with other activities, since you have a lot going on lately. In order to keep your events organized you decide to create a calendar only for work related activities. Name the calendar simply “Work” and make the activities appear in purple color.

2.You have a meeting with your boss today. You want to create an event named “Meeting for work”, the meeting starts at 2:00pm until 4:00pm. Make sure to put it in the “Work” calendar that you created earlier.

3.Set a reminder 10 minutes before the meeting starts, just in case you forget that you have a meeting.

4.You plan to throw a big celebration party for your birthday next year. You want to check what day of the week will your birthday be, while wishing for Friday so everyone can show up! Check the calendar and tell me what day of the week your birthday will be in 2017.

5.You have already created an event named “GUADEC” for the GUADEC 

conference you are attending next month but you forgot the exact date. Can you please search for it and then tell me the date.

6.You don’t want to move across different calendars, creating same events multiple times. So, you try to connect this calendar with google calendar (or any other online account you use).

 

I used Jim Hall’s heat map technique, to summarize my usability test results. I find this to be a great way to see how the users performed on each task. The heat map clarifies how easy or difficult it was for the participant to accomplish a certain task.

  1. Scenario tasks (from the usability test) are arranged in rows.
  2. Test participants (for each tester) are arranged in columns.
  3. The colored blocks represent each tester’s difficulty with each scenario task.

Green blocks represent the ability of the participant to accomplish the tasks with little or no difficulty.

Yellow blocks indicate the tasks that the tester had significant difficulties in accomplishing.

Red blocks indicate that testers experienced extreme difficulty or where testers completed the tasks incorrectly.

Black blocks indicate tasks the tester was unable to complete.

Heatmap
(Click for a better view)

Generally speaking, participants accomplished most of the tasks without difficulties.

Tasks like edit a picture, change the color, enhance, apply a filter on Photos and set an alarm, search for an event on Calendar, seemed to perform really well.

Participants encountered the most difficulties in creating an album in Photos and adding a new calendar, adding a new online account in Calendar. There are two tasks(marked in black) that participants were not able to accomplish, which is a small number considering the number of tasks that were completed successfully.

I ordered the columns starting from beginners(left), to experienced users(right). We can notice the “Cool” rows with lots of green and yellow are dominating the right side of the heat map. This indicates the impact of the previous experience in accomplishing the tasks. Areas that describe the experienced users tasks are more likely to be areas with good usability.
This is a wrap for part 1 of usability test analysis. Next up, I am coming with more details “pulled” from the heat map. What was common between the “cool” rows and why did users find these easy to do? What was common between the “hot” rows? What where the challenges and what went well?

More to come soon!

 

First impressions of GNOME usability testing

gnome-symbolic-icons

I am delighted to have completed usability testing on 10 participants!

Generally speaking, the testing process went really well. There are of course some downsides to it. I’ll go ahead and share an unordered list of some things that went right and some that went wrong:

  • Covering a wide range of users

I’ve had people of different professions, age and computer expertise taking the test. This led me to really interesting and surprising findings which I will share in a future post, explaining in details using heat maps and charts. It was really interesting seeing different people taking different paths to accomplish the same tasks.

  • The right number of testers

Looking back at one of my previous posts where I talked about “Deciding the number of testers”  I explained a chart which assumes that 5+ testers are enough to uncover most of usability issues. I found this to be correct, during the testing sessions. As the number of participants kept growing I encountered less and less surprising results. Their difficulties on certain tasks became almost “predictable”. So 5+ testers it’s really enough, if you don’t want to get into details.

  • The right number of tasks

Just before starting the tests I regretted the fact I had only written 16 tasks, thinking that I could include more, therefore cover more features. After the first test I changed my mind. Considering the fact that I spent some time before the test talking about GNOME and explaining the testing process, also answering some questions and including the follow-up questions, 16 tasks were just enough. It would be very hard to keep the participants concentrated for a longer period of time.

  • Blame it on usability

Even though in my pre-test script I explained to each participant that the tests purpose it is not testing their knowledge but it is testing GNOME’s usability instead, I could see they still felt uncomfortable when they weren’t able to accomplish a certain task immediately. So I had to repeat again that it is okay to not be able to accomplish the task, just blame it on usability😉

  • Thinking out loud and recording

Another point where participants felt uncomfortable was communicating their thoughts. Most of them did it only during the first tasks and then stopped. So I sometimes had to ask questions like “Could you please tell me what are you thinking this moment?”. Other times when I did not want to interrupt the participants, I asked them in the follow-up questions like”What were you thinking during this task?”, “Why did you make that choice?”…

Participants also felt uncomfortable to record their faces or voices during the test, so I only recorded the screen and wrote some notes during each test, so I can refer to them during the analysis.

  • Struggling with the tasks

I am really happy with the fact that the participants found the tasks clear and straightforward and accomplished most of them successfully, but there were some tasks that they all struggled to accomplish. Even though I mentioned in the beginning of the test that I can not help with the tasks, I am here just to observe. Yet they constantly asked for help. I noticed that this is a challenge for others who do usability testing too. In this case I was not an exception. It was really hard not to give hints, specially when they were so close to finish the task.

  • Yes to GNOME!

Except the tasks every participant had to accomplish on Photos and Calendar, I also let them explore GNOME a little bit. For example I let them relaunch the applications when they accidentally closed them, let them search for files(of course I offered help in case they needed, since this Is not part of the test). In the end I asked them about their overall impression of GNOME and if they would use it daily, and they all answered positively. They also asked a lot of open source related questions and some where interested to start contributing to open source projects.

 

So this was just a short first impression of the usability testing I conducted. There is a lot more to cover. I am planning to write more posts, that way I can get into details of the test.

See you soon!

Outreachy talk

outreachy-logo

So this happened🙂
Yesterday I gave a talk about Outreachy to Girls Coding Kosova. Since there is isn’t anyone else from Kosovo who participated in Outreachy previously and they were not really informed about it, I thought I’d share my amazing experience and give some details about the program. I decided to focus more on the application process since that was the “tricky” part when I applied and seemed to be the same for them as well, since they had a lot of questions regarding the application part. I pretended to be applying for the second time and went through the application process step by step. Starting from choosing an organization, choosing a project, contacting mentors and coordinators via e-mail or IRC, making a small contribution etc.Outreachy3

I also talked a little about my project “Usability testing for GNOME”, so they would have an idea on how a project looks like and also walked through the project phases, I think this will come in handy specially to those who decide to apply for Usability testing as well. I tried to explain what is usability and how important it is for the users. Some of them liked the project and even decided participate on the test ^_^T

So, I must say it went great overall🙂

I really hope they have the opportunity to join the amazing experiences that Outreachy has to offer!

Talk-GCK

Final scenario tasks and preparations

c1

Few weeks ago I posted Shaping the scenario tasks, where I shared a “draft” of the scenario tasks and promised to come back with some improvements as the “final version” of these tasks. And here I am, with quite a lot changes and improvements actually.

We decided to elaborate more the scenarios since they did not cover much of each application. We narrowed down the number of applications to two, and started looking them in more details. I also contacted Photos and Calendar, developers and designers, to ask what they’d be interested in having tested. They were really helpful and suggested some new features to test with like: non-destructive editing in Photos, alarms and new edit dialog in Calendar.

There is a lot to cover in both applications but I tried to combine these suggestions and keep the tasks as short and straightforward as possible since the time is limited.

 
Photos:

1. You just got back from your trip to Thailand. You want to show your friends all the pictures that you captured there but you notice that they are all mixed with the other pictures that were previously there. To avoid the confusion you decide to create an album that contains only pictures from your trip and name it “Thailand trip”.

2. As you are showing the pictures of the album you just created, you notice a picture that you’d like to share on your social media accounts. You “favorite” that picture, in order to access it more easily later on.

3.While looking trough all the photos, you notice that two of the photos look very similar, so you decide to delete one of them.

4.After showing your friends all the pictures of the “Thailand trip” album, you revisit your “favorite” picture since you want to edit it before sharing on social media.

5.You start by cropping the picture to make it look smaller.

6.Than you want to play with colors a little bit. You want to make the picture brighter and lower its saturation but leave the contrast as it is.

7.Now you want to enhance your photo. You sharpen it a little bit and leave the denoise as it is.

8.And finally, you choose a filter named “Calistoga” as a final touch.

9.You want to apply these changes, so the photo remains as it is.

10.You really like the way that the picture turned out and you decide to set it as a “Background picture”.

 
Calendar: (I plan to use GNOME Continuous image to test with Calendar)

1.You want to have all your work related activities in your calendar but you don’t want them to be mixed with other activities, since you have a lot going on lately. In order to keep your events organized you decide to create a calendar only for work related activities. Name the calendar simply “Work” and make the activities appear in purple color.

2.You have a meeting with your boss today. You want to create an event named “Meeting for work”, the meeting starts at 2:00pm until 4:00pm. Make sure to put it in the “Work” calendar that you created earlier.

3.Set a reminder 10 minutes before the meeting starts, just in case you forget that you have a meeting.

4.You plan to throw a big celebration party for your birthday next year. You want to check what day of the week will your birthday be, while wishing for Friday so everyone can show up! Check the calendar and tell me what day of the week your birthday will be in 2017.

5.You have already created an event named “GUADEC” for the GUADEC conference you are attending next month but you forgot the exact date. Can you please search for it and then tell me the date.

6.You don’t want to move across different calendars, creating same events multiple times. So, you try to connect this calendar with google calendar (or any other online account you use).

 

Final Preparations

Deciding the number of testers

Before starting any usability test, we should think about how many testers do we need to get good results?

I read an interesting article regarding this, which summarizes that the best results come from testing no more than 5 users and running as many small tests as you can afford.
chart

According to this chart, as you add more users, you learn less because you will keep seeing the same things again and again.

With five testers, you can uncover most of the usability issues but this can happen only if you are iterative in your usability testing, meaning that you do a design, test it, refine the design, test it again, etc.

Since I am not going to test iteratively, I will need more than five users in order to achieve useful results. The goal is to find ten testers.

Setting up the testing environment

I have set up separate test accounts for testers and I also prepopulated each account with the sample files needed.

For example, I downloaded some pictures so that the tester can accomplish tasks like: “You just got back from your trip to Thailand. You want to show your friends all the pictures that you captured there…”, I also created some events so that the tester doesn’t need to create lots of tasks(that would be time consuming): “1.You want to have all your work related activities in your calendar but you don’t want them to be mixed with other activities…”.

I installed an application  to do a “screencast” record of the testing process. As I mentioned earlier, that will come in handy later during my analysis.

Testing yourself

I did a “dry run” of the test, to make sure I know what to do, and that everything is there for the test. This is the last opportunity to identify anything that’s missing on the scenario tasks or in the setup.

Ready for testing! I will update you on how it went in the next post…

Software Freedom Kosova Conference SFK’16 Call for Speakers

sfk

SKF | Software Freedom Kosova is an annual international conference in Kosovo organized to promote free/libre open source software, free culture and open knowledge, now in its 7th edition. It is organized by FLOSSK, a non governmental, not for profit organization, dedicated to promote software freedom and related philosophies.

We are inviting all free and open source software and hardware hackers to submit their talks and workshops for the 7 th edition of SFK to be held in Prishtina, Kosovo, on 21-23 October 2016. Priority topics this year: mobile, web and game development, IoT and artificial intelligence. Conference talks to be held on 22 October, a Saturday, should last no more than 25 minutes, including Q&A. Workshops will be held on 21 and 23 October and may last 1-2 hours. Conference talks should be in English. For consideration please fill out http://goo.gl/forms/HnQrmxB1B4dafd492 We’ll let you know by 15 September. Send questions to info@flossk.org

If in doubt, you are encouraged to submit. Please share this Call for Speakers with your hacker friends.

Thank you!

 
Details:

Date: 21-23 October 2016 (Fri-Sun)

Deadline of CfS: 15 September 2016

Website: http://sfk16.flossk.org

 

Pre-test “setup” and wrap-up interview

gnome_logo_3d_by_ilnanny

As I am preparing for the final usability test, I put together a pre-test script and a wrap-up interview that will help me provide some necessary details to each participant, and also get their feedback. In my pre-test script I will talk about usability testing and GNOME, and afterwards I will be doing the short post-test interview on the overall experience of the testing process, and using GNOME.

Test intro:
Thank you for coming today!

Just like Windows is a desktop system and MacOS is a desktop system, GNOME is a free software desktop system. And you’ll be doing a few tasks using GNOME and a few GNOME applications. I’ll give you the tasks one at a time. We aren’t “testing” you. The test is all about the software. If you have problems with part of the test, that’s okay; that’s exactly what we are trying to find. So don’t feel embarrassed if you have problems doing something and please do not feel pressured by time or anything else. f you can’t figure something out, that is perfectly okay and will still provide us with useful information for the test.  If you have any questions about the tasks I will try to answer them, but the answers and feedback should come directly from you as much as possible so I will avoid anything that would lead you to a specific choice.  Also, I’m going to take notes while you’re doing these tasks. It will help me if you talk out loud when you are doing something, so I can take notes. So if you’re looking for a Print button, just say “I’m looking for a Print button.” And move the mouse to wherever you are looking on the screen, so I can see where you’re looking. It would also be very helpful for me to do a screen-record of the testing process, so that I can go back to that later during my analysis.

Follow-up Questions:

  • What is your overall impression of GNOME?
  • Do you feel familiar with the interface in general?
  • Is there anything about the test that was confusing or bothered you?
  • Which tasks made you feel the most comfortable/uncomfortable?
  • Is there anything else you would like to say about the test?

 

I might also ask some questions on a specific task if I need more information:

  • It looked like you had problems doing __ in the test; what were you looking for, or what would have made that more obvious?
  • What were your expectations regarding … ?
  • Why did you choose this method for …?

Thank you very much for accepting my invitation for this usability test, and for your help on improving GNOME!

 

 
As I mentioned earlier I am planning to do a “screencast” record of the testing process, so that I can go back to that later during my analysis. If you have any suggestions on what software to use for that, please let me know in the comments!

SHAPING THE SCENARIO TASKS

This week we are moving on to Creating the scenario tasks for GNOME programs. After a discussion with Jim Hall(my mentor), Allan and Jakub(GNOME design team),we decided to look back at the usability test results from the last round of Outreachy, and focus on the tasks that the participants struggled to accomplish. For example: Finding the zoom button in Image Viewer (header bar button), changing the month/year in Calendar (header bar buttons), searching (header bar button) and copying in Characters (primary window button), annotating and bookmarking in Evince (header bar menus), and other tasks in Nautilus (several were header bar menus). Re-using these scenario tasks will allow us to compare how the design patterns have improved over time.

So, the focus will be mainly on Image Viewer, Calendar, Evince and Nautilus, since these are the programs that uncovered most of the problems for testers. Allan stated that the obvious thing we’re looking for, is any usability issues people might have with the new design – things that they don’t understand, that don’t work as expected, or which are surprising.

 
I plan to use another account with no changes that could affect the testing process, and add some files that participants might need to accomplish certain tasks. I also plan to launch the applications before they start the test, to avoid any confusion.

 

 
Nautilus:
1.Three days ago, you re-organized your files and you don’t remember where you saved and how you named the presentation you were working on. Please search for it, so you can continue working on it.

2.Files and folders are usually displayed as icons, but you can display them in other ways too. Change how the file manager displays files and folders, to show them as a list.

3.You don’t have your glasses with you, so you aren’t able to read the names of the files and folders very well. Please make the text bigger, so it is easier to read.

4.In the Documents folder, there is a folder named HB. This is a special folder to keep important work related to a project you are working on. Save the location of this folder so you can visit it again in a few weeks when you work on this project again.

 

 
Evince:

1.You want to quickly get to a certain part of the document you are working on, later. Save the location of the title in order to have an easier access to it the next time you are in the document.

2.You want to make a final review of your article. Please adjust your view of the document so you can see two pages at once.

3.After you finish, please save all your changes you made to the document, then close it.

 

Calendar:

1.Your boss just called, and cannot make it to a meeting with you. Your boss asked you to move the meeting to another time. Please create a new meeting for August 17, 2016, from 2:00pm until 4:00pm.

2.One of your colleagues just reminded you about an important conference you need to attend September 01-03, 2016. You already planned a family trip for around that time, so you want to check if you are free to attend the conference. Please check your calendar to see when you have your vacation.

3.You plan to throw a big celebration party for your birthday next year. You want to check what day of the week will your birthday be, while wishing for Friday so everyone can show up!

 

Image viewer:

1.You and your team want to make a birthday surprise for one of your colleagues. You collected many virtual postcards from everyone, and used these to create a funny personalized postcard that you plan to show during a team meeting. You want to review the image before the meeting. (The image has already been opened for you). Please adjust the image and make it bigger until you can read the text in the balloon.

2.It’s time for the show! Please start a slideshow of all the postcards.

 

 
As I mentioned earlier, I reused and modified some of the scenario tasks, and neglected some basic tasks that performed well in previous usability tests such as opening a file, saving a file, etc.

I added some new tasks that I found suitable. For example: I added the first task in Nautilus, (1.Three days ago, you re-organized your files and you don’t remember where you saved, and how you named the presentation you were working on. Please search for it, so you can continue working on it). I thought it would be useful to test more options on the search bar.Search

 

I also added the third task in calendar,(You plan to throw a big celebration party for your birthday next year. You want to check what day of the week will your birthday be, while wishing for Friday so everyone can show up!), I intend to test how intuitive is the view switcher for the participants.cal

I think 12 tasks are enough since the usability test should be 30 to 40 minutes, with some extra time before and after for a brief pre-test “setup” and wrap-up interview(which I will show on a separate post).

This is just a “draft” of the scenario tasks, I will come up with the final version after Jim, Allan and Jakub decide which tasks are more suitable.

Analyzing GNOME design patterns

After learning more about “What is usability,” different ways to test usability, personas, scenarios, and scenario tasks, the research phase is over. We are now starting the usability testing phase which should be really exciting!
Next step is Reviewing GNOME design patterns, before we jump into creating scenario tasks for the final usability test. Throughout this week I spent some time getting familiar with GNOME Human Interface Guidelines(HIG).

 
The primary role of the HIG is to enable people to design high-quality, integrated and consistent applications for the GNOME platform. It also serves a secondary role in helping new GNOME contributors to learn the project’s design goals and approaches. Patterns and user interface elements form the core of the HIG. Together, they are the building blocks for application design.[0]

 
Patterns are the primary elements that make up an application design. Some patterns are highly recommended and others are optional.

 
Core design patterns- Application menus, Dialog windows, Grids, Header bar menus, Header bars, In-app notifications, Lists, Notifications, Primary windows, Search, Selection mode.

Supplementary design patterns- Action bars, Empty placeholders, Info bars, Overload controls, Sidebar lists, View switchers.

 
For more details and appropriate uses of each pattern, I suggest visiting GNOME design patterns page, since they are explained really well there.

 
The goal for the usability test is to examine as many Design Patterns as possible. I managed to create a 7×7 matrix to better understand what design patterns are used across different applications.

Matrix

 

So, we can notice how many patterns are being used in each application:

Nautilus, Software, Web, Music, and Gedit: 5 patterns

Photos: 6 patterns

Maps: 4 patterns

 
We can also notice that the most used patterns are: Application Menus, Primary Windows, Header Bars, and Search.

 
Next week I will be creating a “set” of scenario tasks for these particular applications and some of their features!

 

 

Related links:

https://wiki.gnome.org/Design/HIG [0]
https://developer.gnome.org/hig/stable/
https://developer.gnome.org/hig/stable/patterns.html.en

Scenario tasks

A scenario task is a set of steps that the participant needs to perform to accomplish a goal. Once you’ve figured out what tasks you want to test, you’ll need to formulate some realistic task scenarios for participants to accomplish. In short, a task scenario is the action that you ask the participant to take by providing the necessary details to accomplish that task.

A scenario task should be given to the participant during the usability test and they should represent what these people actually do with that software. Scenario tasks are the best way to bring to light usability issues since you can really see the way the participant is trying to finish the task given by you. These tasks should be created carefully because their formulation determines the reliability and the validity of the overall usability testing.

 
Before I applied for Outreachy internship I had to present a “first contribution”, in my case a usability test with few testers and ten scenario tasks. This test served as a glimpse on how the final usability test result would look like, and also gave me more information on what I should and should not be doing next time while creating scenario tasks and while performing those tasks. I managed to summarize three short tips on how to create better scenario tasks:

Do not give clues
Avoid statements that may end up giving too much information. Instead of saying click“this”, you should leave the participant finish the task in order to see if that feature is intuitive. Also do not force the user to execute that task in a certain way, there are many ways to accomplish a task so let the user’s choose their way to use the interface.

Use the user’s language
Try to adapt to participant’s way of speaking based on their backgrounds and experience using that software. Using terms they do not understand well leads them to confusion and makes them feel uncomfortable.

Keep the scenarios short
The biggest challenge during the test was keeping the testers focused in their tasks. You want to keep the scenarios short with just enough information so you can keep them interested without going overboard with details.

 
In the upcoming weeks you will be able to see these tips “in action” since we will  create more scenario tasks for GNOME applications.