Testing and evaluation

Test and evaluation

Start!

will it work?

PHASE FIVE

index

Tools

how?

why?

Intro

Introduction

Looking at your prototype, what a sense of accomplishment! It has been a long way and now you have developed an actual soon-to-be-product that will disrupt the world!

But, will it?

You will never know until you test it.

Design thinking is a methodolgy and an approach to create innovative, user-centric products. It is a living process that is constantly evolving. This final phase "test and evaluation" is a stage where you purposfully design a testing secnario to reveal areas of improvment and futher research.

Testing leads to failure, and failure leads to understanding.

- Burt Rutan

Why do we test?

+ info

Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi.

Improve the prototype

Observation and feedback gathered during the test will informs the next iterations of prototypes.

You will be able to see if there is a bug in the program or if users confused by the instructions.

Sometimes this means going back to the drawing board.

Learn more about users

Testing give you an opportunity to interact, engage and observe with the users.

This help you to empathise more — often yielding unexpected insights.

Refine the problem statement

Testing may reveal that, not only did you get the solution wrong, but you also framed the problem incorrectly.

Save time and money

It is always faster, easier and cheaper to change, adapt and fix prototype at an early stage.

And if the idea is actually not working at all, it may be better to cut losses.

How to test?

Define Goals and Scope

You won't get useful data if you are asking wrong questions.

Re-visiting your problem statement, consider how you can measure success and identify key questions or knowledge gap you have for the prototype. Remember you are testing the prototype, not the users. The test is to tell you more about your prototype from the user's point of view.

For examples:

- Is the design user-friendly for sight-impaired users?

- Does the prototype attract users from a certain age group?

Who to test with

Using insights from the Empathise stage, take time to understand the key differences in user attitudes or behaviours, that may influence how they understand and interact with the product or service. Make sure that you recruit a sample which represents the whole target audience and covers the extremes where people are more likely to encounter issues or have different opinions.

Conduct the test

During the test session, try as much as possible to recreate the real-life, natural environment where the participants will use the product.

Use active listening and observations to uncover honest feedback.

Seek to understand why a solution does or doesn’t meet their needs.

You may also provide alternative solutions for users to compare against one another.

Consider the logistics

When planning for the testing, take into consideration participants' profile, testing procedure, internet connection, venue and timing.

You want to ensure the test sessions run smoothly with minimal interference. You should aim to have consistent and focused session while retaining some flexibility to probe further on areas of interest.

Iterate and re-test

Analyse data and feedback capture during the test.
Communicate the analysis with relevant team members and make adjustments for the prototype.

Once the prototype is updated/iterated, re-test.

tools

Built it - Break it - Fix it

Wizard of Oz

Feedback Grid

Usability test

Build it- Break it - Fix it

Often used in software development, "Build it - Break it - fix it" aims to stretch and test a prototype under extreme situation and make it fail-proof.

Build it - Turn your concept into a prototype FAST. Don't drag on or wait for more market analysis. Build it, even if it is not perfect

Break it - Once you have your prototype, find someone (users, colleagues) and ask them to break it. Essentially, you want identify the weak points and things that are not working well.

Fix it - Analyse your "broken prototype". Look into things that broke and also things that bent. Use the "5 Why" to find the root cause and re-bulid it.

+ info

Wizard of Oz

+ info

video

No one wants to spend all the money and time to build a full prototype that no one likes. WELL WELL WELL, call on the Wizard of Oz to do the trick for you!

Mostly used to test digital systems, once you decide what function you want test and find a way to fake it (by using tablets, email systems, online chat etc.) Make the user thinks the response is computer-driven, when in fact it’s human-controlled.

Voilà! Now you can observe and collect users responses without a fully functional prototype!

Usability Test

A usability test or user test is to let selected users test your prototype, observe their behaviour and reaction, and identify weaknesses and potentials from the user's point of view.

Usability test is an essential step of product development. The sooner the prototype is put in front of a real user, the earlier and easier you can improve.

There are many types of usability tests, like

+ info

video

Each usability test is differently. The key is to pre-define what questions you want answers for.

sample questions to get you started

Do people enjoy using your product?
Are users able to successfully complete pre-determined tasks?
Does the product match your core target’s expectations?
How easy to use is your product?
Are users pleased with the interface, the content, the process of your product?

Moderated and Unmoderated Test

+ info

Moderated

Unmoderated

Moderated vs. Unmoderated Tests

Moderated Test	Unmoderated Test
Allows for individualized follow-up questions in real-time	Often features a quick turnaround in receiving contributor feedback
Real-time assistance from the moderator	Suitable for obtaining larger sample sizes
Can be conducted remotely or in person	Is self-guided and unmonitored

The primary difference between moderated and unmoderated tests is that with moderated tests you (or a moderator) have to be present to guide the contributors through the tasks. With unmoderated tests, you do not need to be present, and contributors are in their natural environment.

Feedback Grid

+ info

video

A 2x2 Feedback Grid is simple yet effective way to capture the responses and insight from users.

Ask participants four questions about their user experiences with your prototype:

1. What do you like about it? (Likes)
2. What would you change? (Wishes)
3. What questions came to mind during the test? (Questions)
4. Any new ideas? (Ideas)

Once you have collect all the feedback. Analyise them, look for patterns and theme, then take actions and adjust!

Testing and evaluation

More creations to inspire you

PROMOTING ACADEMIC INTEGRITY

ARTICLES

AGRICULTURE DATA

THE OCEAN'S DEPTHS

C2C VOLUNTEER ORIENTATION

LAYOUT ORGANIZATION

TALK ABOUT DYS TEACHER-TEACHER

Transcript

Moderated vs. Unmoderated Tests