How We Collect User Data at Oracle

At Oracle, we believe in getting our user interface designs right. To accomplish this, we ask users about their wants and needs, and we observe them using existing and proposed software. We involve a variety of users, including customers, partners, and end users, depending on the product or flow that we are testing. We bring users into our state-of-the-art usability labs located around the world. Through interactive experiences with our usability professionals, we obtain a clearer picture of any issues with a given design, and we use this information to improve the design. This process is repeated throughout the lifetime of a product.

In this article, I describe typical methods of User Experience (UX) teams in performing data collection from end users, to validate and improve the user experience of Oracle applications. This process has four main phases:

• Engaging the user
• Setting up the environment
• Conducting the usability session
• Providing the results

These steps are part of the Oracle User centered design process depicted in the schematic below:

Engaging the User

We maintain participant databases of individuals who are willing to take part in our usability activities at Oracle. These people have job experience and other qualifications identical to current and future users of Oracle products. Our current databases contain nearly 5000 test candidates globally. They are screened for eligibility on a per test basis and sign non-disclosure agreements before they start the tests.

Recently, while the User Experience (UX) team was reviewing designs of our next release of the applications suite, we received an email from a sales consultant expressing the frustration that a customer was having with an existing user interface.

We invited the customer to participate in our user-centric data collection activities to provide feedback on our existing products while also helping to foster ideas for future designs. The customer agreed to send several participants to take part in a number of product design and development activities, including customer advisory boards, focus groups, and usability sessions.

Setting Up the Environment  

The Oracle Usability Labs were founded in 1994. We currently have 18 usability labs located in 7 locations worldwide: Redwood Shores, CA, Pleasanton, CA, Denver, CO, Burlington, MA; Reading, UK, Hyderabad, India, and Bangalore, India. These labs provide state-of-the-art technology for observing, measuring, and recording usability data, both locally and remotely.

Participant side of Usability lab with one way mirror in Oracle Headquarters, Redwood Shores, California

We set up the appropriate environment for the usability session with the participants. We arranged an office-style usability lab for the customer feedback session, as this type of lab is best suited to conduct usability evaluations, perform guided walkthroughs, and test paper prototypes.

The usability session required a live environment that would not change between participants. We set up the hardware and software for the required environment and trained the usability engineer on the process of refreshing the environment, so that each participant would see identical information. Maintaining an identical environment for each participant helps to ensure that we gather consistent and accurate data.

Conducting the Usability Session

The Oracle user-centered design process supports international standards in usability methods, such as ISO/IEC 25062, which is the Common Industry Format (CIF) for reporting usability test results. Oracle was heavily involved in development of the standard with the National Institute for Standards and Technology (NIST), Boeing, and other Oracle customers and vendors. These standards recommend collection of data pertinent to user effectiveness, efficiency, and satisfaction with the product, in task scenarios similar to how they will actually work with the product. Customer involvement is often critical in producing real world use cases which allow the valid testing of software under development.

Prior to conducting the usability test with the customer, we conducted a pilot test with typical users and benchmark session with a product expert, to estimate the time taken to complete each task, the task difficulty, and other details. Participants were informed that the sessions would be recorded and viewed through a one-way mirror and also viewed remotely using the Web Conferencing application in Oracle Collaboration Suite.

The participants then tested our user interface and provided invaluable information about their experience with our product. One of our usability engineers logged this information using our in-house data logging devices that help us compile the data more efficiently and accurately.

The sessions were observed live in the lab, as well as remotely, by usability engineers, designers, and developers. The sessions were also digitally recorded as high-quality MPEG4 files for distribution to UX team members who could not attend the sessions. The MPEG4 files were also used to review the sessions and note any additional observations that may have been missed during the live test..

Providing the Results

A summary of the results of the sessions were compiled and provided to the development team. The development team implemented some key changes in the software based on the results of the usability sessions. These changes will be available in the next release of the software.  

A detailed report was created and published, providing in-depth analysis of our findings.A version of our test reports in the NIST CIF format was available to the customer under non-disclosure. The customer was thrilled to be engaged in the development process and was excited to see their involvement pay off in increased productivity due to the improvements that we made in the user interface.

For futher information

For further information on the Oracle Usability Labs, contact Jatin Thaker.
For further information on the NIST IUsR project and CIF reports, contact Anna Wichansky.