What Happens in Vegas Doesn’t Always Stay in Vegas!

Gathering Customer Feedback at a User Conference

What is Oracle doing to make sure its new products will work for customers?

Recently, Oracle’s HCM (Human Capital Management) UX team conducted a set of Customer Feedback Sessions on Fusion prototypes at a user conference in Las Vegas . . . an immense undertaking, but well worth the effort! We gathered critical feedback and, needless to say, the information gleaned in Vegas did not stay in Vegas. The valuable feedback we received is now being fastidiously incorporated into the next iterations of Fusion application designs.

Going to the Customer

Why did the UX team decide to gather feedback at a user conference? We felt it was an ideal opportunity to meet the customer on neutral ground, take advantage of a constellation of important profiles, create a buzz about Oracle’s latest design innovations, and learn firsthand what HR professionals want in their next release of Oracle Applications. 

Customer Feedback Session

Oracle’s UX team has established a number of Oracle-specific usability methods for gathering critical customer feedback. The particular Oracle methodology selected for the user conference was the Customer Feedback Session

In a nutshell, Customer Feedback Sessions are the equivalent of a Pepsi taste test . . . it’s a usability activity in which a customer who fits the targeted user profile completes tasks in a prototype by clicking through it and speaking aloud. 

Customer Feedback Sessions are flexible usability activities that can be conducted outside of a usability lab. Not much equipment is required: just a laptop, monitor, and mouse. Feedback obtained through these sessions helps a designer learn how to improve the prototype design in the next iterations. The designer can answer the following questions:

• Can the user navigate naturally through the prototype?
• Does the user understand how to start and complete tasks?
• Does the user understand the terminology used in the prototype?
• Can the user discover features?
• Does the user interpret features in the way intended?
• Are you giving users the right content they need on a page?
• What does the user like about the prototype? What are the user’s main points?
• What is the user’s overall impression of the layout?

Organizing the Team

The HCM UX team started organizing the Customer Feedback Sessions months prior to the conference. A number of i’s needed to be dotted and t’s needed to be crossed to guarantee that the lab would be functional by the time we got to the conference. 
           
First and foremost, we needed a space in which to conduct the tests. Thanks to the conference staff, we got a great room just a few minutes walk from the main conference.
           
The next step was to choose the right prototypes to test. The UX team worked with HCM Senior Management to determine which prototypes required customer feedback.
Prototypes from each functional area in HCM were selected to ensure a comprehensive and complete picture of design feedback across the HCM application suite. 
           
Next, members from the product teams were assigned to the project. For each customer Feedback Session, we needed a moderator as well as a note taker. The UX team planned a number of comprehensive training sessions to make sure that each person involved understood the testing methodology.
           
The HCM UX team provided the following comprehensive workshops to product managers:

• Scenario and Task Writing Workshop – This detailed workshop explained how to expertly write tasks that the user would complete at the sessions. The UX team explained the importance of writing realistic, goal-oriented scenarios that encouraged the participants to navigate through the prototype. 
• Moderating and Notetaking Skills Workshop – This second workshop focused on how to conduct the Customer Feedback Sessions. The UX team outlined to moderators and notetakers how to actively listen to the customer comments and watch how they click, regardless of whether they succeed or fail, when they click through the prototype. We showed how a moderator would deal with a frustrated participant or prod a customer for further information about a specific task through various role-playing exercises.

Doing a Test Run: the Pilot Session

After the training, the moderators were required to put their training into action in a pilot session. All team members, including the User Experience buddy, were required to attend the pilot session. After each pilot session, execution was discussed by the team and revisions were made as required. Additionally, the User Experience buddy provided constructive feedback about moderating skills.

Planning for the Conference Customer Feedback Sessions

Obviously, setting up a temporary usability lab at a conference center requires planning and coordination. The focus was not going to be logistics, but watching, listening, and learning from the customers as they navigated through the prototype. 
           
The UX team had to transfer the stark conference space into a temporary usability lab.  The UX team worked with Oracle marketing to get banners, tablecloths, and posters. Additionally, the UX team worked with conference organizers to secure furniture, monitors, keyboards, and mouse devices for each station. Refreshments were coordinated with the hotel banquet services.

A barren hotel banquet room was miraculously transformed into a functioning lab where customers felt comfortable sharing their feedback, as shown in the following diagram and photograph. 

Temporary Usability Lab Layout

Waiting Area

A Customer Feedback Session in Progress

Recruiting Customers

The most critical planning piece was the actual recruitment of customers. After all, they were the reason we were organizing these tests. The UX team sent targeted e-mails to conference attendees asking whether they were interested in providing feedback. We heard back from the interested conference attendees, booked their usability sessions, and informed them where the sessions were being held; we also briefly explained the manner in which we would be gathering their feedback. 

Time for the Sessions

Customers showed up ten minutes before their session to check in and were whisked away from the hustle and bustle of the casino into the quiet waiting area in our temporary usability lab. To ensure that each customer was relaxed and in the right frame of mind to provide reliable feedback, we provided him or her with refreshments.    
           
The moderator took the relaxed customer to the designated testing station. At the station, the customer was introduced to the product team and provided with a high-level overview of how the Customer Feedback Session would be run. The following statement was typically made by each moderator:

Please understand that we are NOT TESTING YOU, nor your abilities. We are testing the user interface, and if anything is unclear, difficult, or frustrating, then we have some work to do to make it more clear and intuitive.

The moderator then asked whether the customer had any questions or comments.  When the recording started, the moderator asked the participant to read aloud the first task. Now it was time for the product team to watch, listen, and learn from the customer. The product team interrupted only if they wanted to gather further feedback on the customer’s impression of the design, or needed to encourage the user to speak aloud. To encourage this dialogue, oftentimes the moderators were heard to say the following:  

• What are you thinking here?  
• What are you trying to do?

The sessions were limited to an hour. After completion, some final questions were asked of the participant:  

• Can you tell me your top likes and dislikes of the designs you saw today?
• How does this compare to what you are currently using?
  

After all the feedback was obtained, the product team thanked the customers for their valuable time and gave them a complimentary gift. The product team took a short breather, then waited for the next Customer. Typically, eight to ten customers provided feedback on a particular prototype.  

Documenting Usability Findings

A few days after the conference, the teams were requested to outline and submit their high-level findings. Finally, two weeks after the conference an official usability report was required. A sample report was sent to the teams so that they understood how they would need to compile a report and present their findings as well as corresponding customer design recommendations. 

Finally, after all reports were submitted, the HCM UX team prepared a complete report detailing the issues and design recommendations. The high-level issues that were found in the report were escalated to senior management. 

What the Customers Thought

The Product team and the UX team both found the experience invaluable. So much relevant feedback was provided from customers that many product teams needed to take a step back and rethink their designs.      

What did the customers think of the experience—of spending an hour of their time walking through a Fusion prototype?

Many thanked us and told us they were pleased with the experience. Almost every customer also filled out a brief survey. Amazingly, every person who filled out the survey said that they enjoyed the experience. Even more impressive, each participant said that he or she would like to be involved in another session of this sort. Here is some of the positive feedback we received:

• "I liked being able to influence items that will affect my company in the future. I liked being able to see that Fusion will work for us when we switch.”
• "Giving feedback is good, seeing the prototype in action is outstanding.”
• "I feel like our needs are being listened to - we're being included in the solution.”
• "I like the face-to-face feedback session rather than on the phone.”
• "Great to see how new products evolve and ability of users to give feedback.”
• "Have more of these!”

Summary

Conducting Customer Feedback Sessions at the user conference required a great deal of planning, coordination, and precision in execution. However, the overall gain for all parties involved is immense. Just a few of the benefits are:

• Hands-on opportunity for customers to interact with designs.
• Face-to-face experience instead of remote or phone sessions.
• Great occasion to build a design relationship with customers.
• Fantastic opportunity to introduce the User Centered Design process.  
• Efficient and cost-effective method of gathering customer feedback data:
• No additional travel cost for customers
• Only an hour of time invested by the customer
• Opportunity to collect feedback on multiple prototypes at once.
• Opportunity to reach out to users from different locations (Europe, US, Asia, and so on.)

Breaking Vegas Tradition

When it comes to Oracle Customer Feedback Sessions, the details of a busy week in Vegas are not going to be swept under the carpet, as they might be after some bachelor or bachelorette party. No, in this case, what happened in Vegas has been chronicled and documented, and will be thoroughly exploited!

The feedback from the usability sessions held at the conference was recorded, and detailed notes were taken on the customer’s interpretation of the design. The UX team reviewed the feedback from each customer and compared it against that of others. The UX team is now working diligently on the next iteration of Fusion prototypes that incorporates this valuable customer feedback. The HCM UX team is diligently working to set up shop at the next HCM user conference. After all, if you can’t come to an Oracle usability lab, fear not, we’ll come to you! The HCM UX team is currently looking to plan usability sessions at major user conferences in both the U.S. and the U.K. If you are interested in being involved, please contact us by filling out the form at http://ui.oracle.com/getinvolved.htm.