IBM Research Is Using Augmented Reality to Improve Maintenance and Troubleshooting
Proof of Concept app allows maintentance techs to lerverage collective knowledge.
By Oded Dubovsky08/03/2020
The past year has seen an enormous increase in the use of online meetings and video chats and a massive jump in the use of mobile devices for remote consultation and assistance. With travel bans in place and limits on how we interact, people are finding new and creative ways to do things that don’t involve a personal visit to a physical location.
But business is not just about people working remotely from home and shopping online. Industries that rely on machinery are also looking for new ways to improve productivity by providing remote guidance to enable less experienced technicians to conduct maintenance and troubleshooting operations.
Remote Guidance Offers a Solution
My team of IBM researchers in Haifa, Israel, is using augmented reality (AR) in the form of visual operation guidance to help people remotely access knowledge and expertise through 3-D navigation, object recognition and deep learning. Visual Operation Guidance can provide easy-to-use, step-by-step procedures that guide technicians of any experience level at the physical site.
The platform improves the efficiency of technical staff and preserves their knowledge, making it reusable over time. It includes three types of components: peer guidance that allows a technician to get help from a remote expert, self-guidance for independent work, and a “studio” that can be used to create AR experiences for self-guidance. All three options make the data and knowledge visually easy to access and easy to follow.
Self-guidance is achieved by using object recognition to add a layer of annotations on top of the video on the technician’s mobile device. For peer guidance, these annotations are sent to the technician’s device from a remote expert.
In a recent pilot, IBM Research partnered with a large machine manufacturer using technology to support field maintenance technicians and troubleshooting experts. The client typically has a technician residing on site with the machine for six months to a year from the date of purchase. Then they move to a support model where the technician is available on call, and one person must provide this service to a number of business sites. This is complicated by varying levels of experience and areas of specialization among technicians and the challenges associated with using video chat or online images to connect senior technicians with technicians in the field.
An AR solution would help make the process more efficient by reducing the reliance on senior technicians or lengthy manuals to ensure a maintenance procedure was performed correctly.
The client compared the IBM Research AR remote support technology to off-the-shelf products they had already tried and saw the distinct advantages afforded by a customized asset. They also noted the researchers’ strong expertise in AR and ability to address their unique challenges. The results of the pilot showed that the client was able to significantly increase productivity and improve the agility of their support team using the IBM application.
Measurable Results Showed a Boost in Efficiency
Initially, we visited the client’s facility, documented the top recurring issues to be addressed and helped the client define KPIs so they had a measurable way to assess the results, including cost, time to resolution and more.
As part of the pilot, we addressed two main use cases:
- Periodic maintenance
For periodic maintenance, the client’s technicians would have to comb through a pile of books and follow instructions to ensure all parts were in order and the machine was tuned correctly. When it came to troubleshooting, where the problem and its cause are generally unknown, the client needed experienced engineers or technicians to analyze the issue. This often took a great deal of time and involved travel to the site.
The duration of the pilot was three months from definition to measurable results. Partnering with the client, we measured the completion times for different maintenance tasks and activities that were carried out using both self-guidance and remote peer guidance technology. To evaluate the use of peer guidance technology, junior engineers were asked to complete tasks on their own.
In one case, an inexperienced technician who had never used the machine before was able to complete the assigned task in approximately the same amount of time as it took experienced engineers. This was a welcome surprise for both the client and our team.
The pilot measurements proved that for some maintenance tasks, the IBM self-guidance technology allowed the more experienced technicians to complete the instruction sequence in half the time. For the troubleshooting tasks, the peer-guidance technology allowed the less experienced technicians to solve the problem in almost the same amount of time and efficiency as the more experienced technicians.
Other interesting results from the pilot showed that technicians found the technology to be simple to use, and less experienced technicians were more accepting of the new system than more experienced ones.
Extending the Technology to Other Industries
AR-assisted self-guidance and peer guidance can be used to boost the efficiency of technicians in any number of industries, share and preserve the knowledge of an aging workforce, reduce the need for travel and even help the owners of new home appliances get up to speed.
The research team also produced a software development kit for building similar self-guidance applications for the iOS and Android platforms. Developers can easily adjust the provided skeleton to their needs, specializing the UI, annotations and screen-to-screen flow of instructions.
Oded Dubovsky is the manager for Computer Vision and Augmented Reality at IBM Research–Haifa.