Background

Our brain stores information in a way that is easy to recall, such as visual images, body memories, or a scenario. There have more things that we instantly associate with our memories. Hence, that connection makes the information more memorable and easy to recollect. In this age, people have increasingly relied on the internet and personal devices for self-directed learning, such as smartphones, tablets, and laptops. Duolingo, Mondly, and Busuu have made tremendous in language learning. They provide easy-to-understand lessons about foods, people, objects, and conversations that most users are familiar with. The learning experiences help users associate them with foreign language words. Even though they make the language learning process much easier than it used to be, we still agree that learning a language effectively is all about placing yourself in the environment. We recognized this gap and used augmented reality to engage and motivate users to build their knowledge on real-world observations. Users can practice and learn languages via an understanding of different cultures and exploration of attractions around the world.

Challenge

How do we engage with the individual learning pattern and help learners to practice everyday phrases?

Solution

Make connections to learn basic vocabulary from observations and use full sentences in our everyday environment.

A glimpse at behind the scene

Process

Competitive Audit, User Research, Prototype, UI Design, Usability Testing

Research

The Product's Goal

Focus on learning the necessary vocabulary in complete sentences by observing your context using augmented reality. For listening, reading, writing, and speaking, the map mode provides users with daily phrases and memory links. Create fun and immersive learning environment that keeps motivation high.

Technology Research

Given the technical challenges we face, we need to study current computer vision techniques and the feasibility of implementing this application.

We summarize the following requirements:

- The app should be able to recognize real world objects using camera and create labels in different languages.

- The object recognition and label generating should be in real time as the user scans their surroundings by camera.

- The location should connect to the real world map and display a corresponding introduction and simple dialogue.

Open source object recognition API

YOLO, TensorFlow, ML Kit, Google Map Platform

Competitor Analysis

Based on a brief analysis of competitors. Duolingo, Mondly, and Tripadvisor became our references for competitors and products. I'm focused on enhancing the learning patterns of everyday phrases using augmented reality and creating a product that helps users learn in their surroundings.

- Enhance learning pattern with daily phrases.

- Observation using augmented reality.

- Connect our memories with travel experiences.

Users

There has three main users groups, namely language learners, mobile users and travelers.

Persona

Understanding multilingual learners and users who are the first time learning a foreign language.

Takeaways

Motivation

- Need to learn a new language when they move to a place where most people speak in a different language.

- Already in a main languages course and wants more resource to memorize daily phrase.

Techniques

- Want to find more chances to practice in daily life.

Thoughts

- Want to communicate when they are traveling.

- Set goals and improve myself step by step.

Frustration

- There is no one to practice the language with.

- Learning is not restricted by location.

Design

Before designing, we decided to start with a storyboard and diagram what the Multi-Multi experience for our users. People are motivated to learn a new language based on work-related goals, migrate to a new environment, want to learn more about a culture or a new language.

Storyboard

1. One day, when users study and practice languages. 2. They think “I want to speak in German when traveling to Switzerland." Before I go traveling, how I will communicate when I go to restaurants, hotels, museums, or events? 3. Users open the “Multi-Multi” application. 4. Search the place they want to visit on the map mode. The user finds a "Zurich ice cream shop". There is a dialogue on the screen showing how to order ice cream in German.

5. The user turns on the camera using augmented reality and learns the vocabulary of the objects around them 6. After a few days, the user remembers more vocabulary and associates the conversations they learned with places on the map. 7. Users practice language by exploring the world and associating with scenes to build their language skills. 8. Users learn language through daily observation and enjoy the fun of exploring the world of the language they are learning.

User Flow & Information Architecture

User flow

We wireframe the following task flows: registration, observing mode, exploring map mode, dashboard, collection, and set language preferences.

Information Architecture

Here I work on designing a responsive website and mobile app. I use a Multi-Multi sitemap to guide the organization of each screen design to ensure a coherent experience across devices. The purpose of the user flow is to provide the user with appropriate additional material, link the user's memory with the language, and be motivated by more local cultures. To ensure learning goals, we annotated endpoints in the information architecture for users to learn vocabulary, dialogue, reading and listening in observation patterns and exploration maps. Next, we will wireframe several layouts for users to test which kind of layout provides a better learning experience.

Prototype

Paper Wireframes

Paper wireframes help organize the main layout, interactions, user flow, and UI design. I can quickly iterate to find a suitable layout, then think about the main user flow.

Low Fidelity Prototype

In the low-fidelity prototype, I start with the user flow, the goal is to make sure the user can smoothly switch between each feature. In mid-fidelity prototypes, I focus on details of functionality and interaction behavior. Then go for the first usability study.

Usability Study

I conducted two rounds of usability studies. Findings from the first study helped guide the designs from wireframes to mockups. The second study used a high-fidelity prototype and revealed what aspects of the mockups needed refining.

1. Sign in and onboarding
2. Home screen and explore the app
3. Search on Map
4. Observe with AR
5. Make notes
6. Learning pattern

Round 1 - Medium Fidelity Prototype

Insight

Onboarding and user flow

- Prototype should be made more efficient or the task should be a little more detailed.
- Onboarding should to get understanding of users' preference, such as choosing languages, practice frequency and learning purpose.
- Switching between functions is confusing.

Search and Observe functions

- For a great searching and observing experience, Search on the map and Observe with AR should put on the homepage.
- Could have some instructions about the observe with AR, add vocabulary and conversation to the collections before start the function.
- Let users set their favorite tabs in addition to the default tabs.

Learning materials and others

- Finding Round 1 was difficult with this prototype, hence all the following tasks became confusing. But main functions still work.
- Practice with attractions help users to connect their memories.
- Whether Offer more tips of organize notes and learning.

Round 2 - High Fidelity Prototype

The second round will focus on future product trends

Insight

Learning materials and Languages

- Whether add more Materials on current functions?
Search on the map- Reading, Speaking
Observe with AR - Vocabulary
- Whether to provide material according to the learning pattern.

Interaction

- Create a new page for the new language when the user switches other languages during the search and watch functions.

High Fidelity Prototype

Here is the multi-multi prototype on Figma.

Main Features

Get new vocabulary by observing with Augmented Reality

Learn daily phrases by searching on the map

Read introductions and practice conversations based on attractions

Make your notes and associate with your memories

Cross platform

Design System

Takeaways

Next Steps

Function and official event

- Friend mode, social mode or hold an event for users to find and search using the camera.

Usability testing

- Follow-up usability testing of new websites and applications.
- Conduct another round of usability studies for users who speak different languages to validate whether pain points experienced by users have been effectively addressed.

Update content

- Update the application according to user needs and add more user learning materials, such as basic conversations, vocabulary cards, event articles or introductions to sights, etc.