9 Tips and Steps on How to Approach this Challenge
Usability testing by itself is not an easy task. As a usability expert and a designer, you spend a lot of time to understand the challenge, create hypotheses, and prepare for the study. And during the study, you try to make the participants feel comfortable and at ease, while, in reality, you are trying to get into their deep thoughts and emotions to understand what they really feel about a certain project or a process. It is important to understand the people, their mentality, and mindset in order to create a comfortable setting and get the most valuable outcomes.
So, conducting a usability test in a different country, with participants from different cultural, educational, and even linguistic backgrounds, can turn out to be even more challenging, as it includes much more research into cultural traits, nuances, and etiquette. It also includes some extra steps of preparation. The following tips will guide you through preparing for and conducting a usability test in an international setting, given you have the usability testing basics covered.
1 – INITIAL RESEARCH
The first step (as when approaching any project) is research. It is important to understand the culture of the country where you are planning to conduct a usability test: what is the official language; what are the main cultural traits and general mindset, etc. If you are doing a test for a website, for example, you may want to look into the existing norms in that country, as it will give clarity to the participants’ responses.
Example: Once, our team conducted a usability study for a website in South Korea. The website was quite simple and minimalistic – nothing unusual for western cultures. However, for Koreans it was “foreign,” “unfamiliar,” and even “Google-like,” given that in Korea they don’t use Google but instead Naver, which covers their search, maps, community forums, etc. They are mostly used to websites with a lot of text, images, and information. Knowing this allowed us to understand better why we received such feedback.
You would need to determine whether you can conduct the test in English, for example, or use an interpreter. You may also need to look into country’s legal restrictions for your product or service. This will also be important when you ask the participants to sign the NDAs or other legal paperwork you may need, as well as the type of questions you can ask or you should avoid.
2 – FINDING A FACILITATOR
It is generally a good idea to find a supporting party from the host country. If you work in an international company, collaborate with the local teams (e.g. market research, experience design), if applicable.
Alternatively, find a market research agency that may help with selecting participants, providing the space, an interpreter, and general facilitation. In addition, having someone the participants can related to, can help to make the interaction smoother.
https://www.twowaymirrors.com/glass/
Example: In one research that our team had to conduct in Poland, a local market research agency supported us by providing the proper space for usability testing (with a double-sided mirror), selecting the participants using our criteria and screener, taking care of the legal paperwork that the participants had to sign, as well as providing a very experienced interpreter. This collaboration greatly facilitated our experience.
3 – SELECTING PARTICIPANTS
Screener: When designing a screener, one important thing to keep in mind is the cultural trait of providing feedback. In some cultures, people are more submissive and not used to sharing their opinions; so, the usability testing may be more challenging, as their responses will generally be positive or neutral but without constructive feedback. So, in your screener, ask questions that require feedback as an answer.
Another thing to consider is the language. At times, the participants may exaggerate their language skills, if English, say, is required. In this case, test their language skills during the screening process.
Participant selection: Schedule interviews way in advance, as the scheduling will be challenging on distance. Here as well, the local facilitator can greatly help. Schedule extra participants in case of no-shows.
4 – DRAFTING A DISCUSSION AND/OR OBSERVATION GUIDE
Draft a discussion guide and make sure to run it by someone familiar with the host country customs.
I don’t believe you need a translation of the guide, unless some of your colleagues in the host country may need that. Make sure to have extra copies of the guide with you to give to the observers and, most, importantly, to the interpreter to review before the interviews.
5 – SELECTING AN INTERPRETER
A good interpreter will make the sessions go smoothly and be extremely fruitful; so, from my perspective, it is worth the time and investment to find a good one.
The interpreters can translate the conversation consecutively or simultaneously. In the former, they wait for the participant to provide feedback and translate afterwards; in the latter, they whisper the translation into your ear, as the participant provides feedback. In my experience, the simultaneous translation was better, as it didn’t slow down the conversation, and I was able to follow the participants’ facial and gestural expression along with their feedback.
It is important to make sure the interpreter translates the conversation fully without paraphrasing.
https://nativetrans.org/article/view/4
Example: In one of the tests we conducted, the interpreter was translating the conversation as a summary of what was said, and we could clearly see that a lot of the conversation details and nuances were lost. The interpreter analyzed the feedback in her own way and, basically, translated her own interpretation. This hindered our study greatly. In this case as well the simultaneous translation is better, as it doesn’t allow much time for translation analysis.
A good interpreter can also provide an explanation of some cultural nuances between the sessions.
6 – SELECTING THE SPACE
You must either find a room to set up face-to-face usability tests or, in case of doing a contextual inquiry, set up the conversations in the participants’ proper environments. In case of the latter, make sure to structure the study in such a way that the participants properly dedicate time to the study and enough space for the moderator and all the observers.
Example: Once, we conducted a study in small kiosk shops in Italy, which was quite challenging given the sizes of the shops, as well as the store traffic. The shop owners couldn’t stop their activities, so our research took much longer with pauses. The solution in this type of limitation may be to select time slots with the least amount of activity, if that is not a factor for the study.
7 – PRE-STUDY PREPARATION
It goes without saying, but reserve your flights and hotels as soon as you lock the dates of the study and recruit participants. It’s quite challenging to schedule and reschedule users, find or organize locations; so, you wouldn’t want to reschedule all of that because of reservation hiccups.
8 – CONDUCTING THE STUDY
Congratulations! At this point you have already planned and scheduled everything. You have arrived to your study location. Now, this is where your research will prove to be useful. Remember that the most important in conducting a usability test is to get the participants comfortable. You must know and conform to the nuances of their behavior and cultural traits.
Example: In South Korea (and some other cultures), it is more common to have handshakes between men. Women, however, may not be comfortable in shaking hands, but will instead bow. And, in general, bowing can accompany handshakes, as a sign of respect. So, complying to their cultural specificities will make the participants feel more welcome. In addition, people always appreciate if you say simple words, such as hello or thank you, in their language (if you can learn that in advance). It creates a more positive and open disposition.
Setup: If you have a usability testing room reserved with a double-sided mirror, in my opinion the most optimal sitting position is for you, as the moderator, to sit between the interpreter and the participant in such a way that the interpreter can comfortably whisper the translation into your ear, but also address the participant. If the table is rectangular (or square), it may help to sit around the corner, while the participant can face both you and the interpreter. This may, however, depend quite a lot on what you are testing.
Have the rest of the observers sit on the other side of the mirror. In simple settings (without the double-sided mirror), an alternative is to create a broadcasting of the conversation into another room (say, using Google Hangouts).
Recording: Always double check whether 1 – recording is legally acceptable in the particular country, customs, and project environment, 2 – if the participant is ok with that. In majority of the cases, you should either have them sign a document that they wouldn’t mind you recording the conversation (whether with audio or video), or record their consent itself.
In-between sessions: As in any usability study, take a moment to reconvene with the observers and the interpreter to review the summary notes. It is important to do so while the feedback is fresh. In addition, the interpreter (and the observers who are from the same cultural background as the participant) can add valuable insights about the behavior and some feedback details.
Participant rewards: You may wonder how to reward the participants in each country. The local facilitator can help with this. Alternatively, do a bit of research to understand the value, say, of $10 in the host country. Some general options include prepaid Visa gift cards that the participants can use anywhere, vouchers, and, even, cash, if that is allowed. You may consider using a different currency, such as gift products, recharging cell phone minutes, etc. This fully depends on your budget, product, location, and what the participants may appreciate in that particular country. If the online research doesn’t yield to much information about the options for rewards, contacting local banks may help.
Example: In Serbia, we provided around $25 vouchers (2500rsd), which we got from the bank (with an additional fee). This seemed to have been the most optimal way for the participants to use their reward. It also seemed to have been a good enough incentive for the participants to be interested in the study.
9 – Post-study
After you complete the study, it is good practice to ask all the observers to share their notes and documentation. Create as detailed summaries as you can. Even though a lot of modern methodologies do not encourage heavy documentation, I have greatly benefited from having the detailed description of the location, research, and results. In addition, you may consider translating them if you need to share with the local teams.
A designer ethical note: be sure to include (and to ask) only the information that is absolutely required for the research to keep the participants’ privacy. If the last names, for examples, are not relevant, it is not important to record them. In addition, share this information only with people you absolutely need. As designers, we do have a responsibility to support the privacy of our usability test participants.
It’s quite a big endeavor to conduct usability testing internationally. However, it is also very rewarding, since not only you gather the essential feedback, but also open up your perspective to cultural differences that you can always bring into your designs and solutions. Being culturally open, aware, and thoroughly prepared will guarantee that your study will go smoothly with very fruitful outcomes. I would love to hear about your experience of dealing with cultural differences, and how has that affected your perspective as a designer. Send a note to mariam[at]yllwu[dot]com.
