INTRODUCTION
Digital devices, in combination with mobile applications, have been revolutionary for putting information in the palm of a worker’s hand. Mobile email on smartphones was the first example of technology that changed the way workers collaborated and exchanged information. Where there has been a gap for mobile workers is getting contextually-relevant visual information in real time from the mobile device. Images and videos accessed through the smartphone or tablet are helpful, but still extremely limited for providing highly targeted information based on time-of-day, location, thing or product data, and other occupationally-relevant business process and environmental information.
A new technology called augmented reality (AR) is now available to address these limitations.AR is the overlay of digital information, including text, images, and videos, onto a real-time view of the physical world through a camera-enabled device, such as a smartphone or tablet. AR is taking information that is found in a variety of applications, such as maintenance records, inspection and repair manuals,and engineering specifications, compiling it, and displaying it on the physical features of the thing or machine. Moving the smartphone or tablet around the object to show different views reveals new information related to the object’s features.
THE POWER OF IoT PLATFORMS FOR BUILDING AR APPLICATIONS
AR applications can become even more powerful by augmenting them with data generated by another set of technologies, revolutionizing the marketplace called the Internet of Things (IoT). The IoT is the sensorization and digitization of the physical world, enabling not only its monitoring, but also using the collected thing data to predict machine outcomes, develop better products, understand usage to create new services, and plan more efficient operations and business processes. The IoT is driving many enterprises to completely re-evaluate how they design, produce, sell, and support their products. ABI Research expects that more than 31 billion “things” will be connected by 2021. Combining AR and the IoT allows employees, such as field maintenance workers, to access vital machine operating statistics to assist in maintenance decisions, or an assembly worker to visually receive pick and place instructions and alerts if tools are working incorrectly or are out of tolerance. But how does an enterprise benefit from both of these technologies? It requires building both AR and IoT applications. This is not an easy endeavor due to myriad technology and supplier options. This white paper dives into the topic of AR and the IoT, exploring the benefits of IoT-assisted AR applications. It will contrast these benefits with the challenges of building IoT and AR applications. The white paper will introduce IoT platforms, their benefits for building IoT and AR applications, and key capabilities to consider. It will conclude with a case study, providing insight on the benefits one company experienced using IoT-enhanced AR applications.
AR APPLICATION MARKETS AND BENEFITS
Key AR ecosystem components include device hardware, an OS open to third-party application development, and a platform for software, as well as content creation and management. The standard device hardware for viewing AR applications has predominantly been smartphones and tablets due to their wide availability and exceptional compute power. The newer devices expected to accelerate enterprise AR adoption because they allow hands-free access to information are smart glasses, HMDs, and other digital eyewear solutions from companies such as Microsoft, ODG, Epson, and Vuzix. ABI Research expects head-worn devices to make up 90% of the enterprise AR device installed base by 2021.
While AR applications can be built using existing IT resources and traditional mobile application development tools, the majority are built by companies that offer AR SDKs. The top AR platform providers are Vuforia (acquired by PTC), Metaio (acquired by Apple), Wikitude, and Blippar (which acquired Layar). Each is different in the devices they support and the degree of sophistication for applying fixed or floating data displays onto real-world objects. But they are similar in one respect, which is the ability to tie AR applications into backend systems, policies, and procedures. This capability is a critical value-add for enterprise markets that want to leverage their own data in AR applications.
Up to this point, AR has been popularized by consumer applications oriented around advertising, entertainment, and gaming due to the limited availability of industrial AR tools. However, enterprises are trialing AR applications today and are expected to be the primary users of AR applications through 2021 as the introduction of new solutions removes barriers to adoption by harnessing the power of these technologies. AR provides enterprises with the following benefits:
Data Reuse
AR applications allow more efficient reuse of existing enterprise application data. Enterprise application data can include maintenance records, service tickets, and usage logs—any combination of which can be tapped into to decorate an AR experience that blends the physical and digital worlds. Using these data in AR applications improves its value because they are delivered in a visually relevant way specific to the machine, thing, or object to which the data apply. The advantage for the industrial enterprise is that a lot of these data are readily accessible via 3D/CAD software and can be leveraged for better occlusion detection, which determines how physical and digital objects fit and move together, due to the granularity and depth of (CAD model) data.
Data Contextualization
Second, AR applications can be used to better contextualize information across a range of industries and job functions. In manufacturing,this could mean giving linemen and plant personnel the ability to abstract backend systems information for 3D product navigation,step-by-step work instructions, and remote visual guidance; for design engineers, it might be the ability to accelerate the product development process without building expensive models and iterative testing procedures; for field service professionals, it is a means of getting machine data from an asset in need of attention to the service professionals that provide the fix. The common thread is that data contextualization derived from AR applications can help optimize operations measured by improvements in speed, efficiency, and quality.
Work Error Reduction
AR applications can reduce errors made by workers in various occupational tasks. Whether the error is caused by too little information, too much information, or poor quality information, AR applications place the right information and instructions at the location of the work task.
Even simple AR applications, such as those that enable glance able work instructions, have shown an improvement in first-time fix rates and reductions in audit-related errors (e.g., via photo work logs) ranging anywhere from 5% to 20%. In some cases, AR applications can save companies hundreds of thousands of dollars in a single day, as happened when a service technician identified with his AR application that a pump flagged for replacement was mislabeled.
Workforce Multiplier
AR applications are a workforce multiplier in two ways. First, training resources are more effective and efficient. Training hours can be reduced because AR applications can include video instructions on repair/replacement tasks overlaid on the machines themselves.
Second, workers can do more by themselves versus sending multiple workers or truck rolls to address the same problem or work task. These benefits are especially relevant in emerging markets where skilled and trained workers are scarce and language barriers limit the effectiveness of training time.
Safety
AR applications can also improve the safety of a work task. For instance, AR applications can include not only written instructions for removing or replacing a part on a machine, but can also include visual instructions. This assistance avoids unnecessary and possibly unsafe interaction with the machine or object.
All of these benefits are magnified when IoT data augments the AR application. For instance, operational vibration data of a generator may detect an installation issue that can be addressed at the same time a faulty bearing is replaced. Another example is partial discharge monitoring data from utility substation transformers, which can be used in an AR application that directs a utility technician to the right cabling components in need of repair or replacement. A third example is to use recommendations from analytics on IoT data to avoid certain machine components during an inspection by leveraging AR to identify internal pressure conditions or dangerous electrical charge build-up. A final example is how greater sensorization of a machine or nearby environment displayed in an AR application can improve work safety conditions for personnel in occupations such as fire and environmental protection.
CHALLENGES OF BUILDING AR APPLICATIONS ENHANCED BY IoT DATA
Even before IoT data are considered, AR application development does have its challenges. The first and most significant of these is content creation, which requires a certain degree of domain-level expertise to determine what to augment, and where to get content from, in addition to the lead time required to do so. While using traditional tools to create a piece of content that is meaningful to interact with can take about a week, conservatively, the need for design experts/solution architects (i.e., multiple people with special and specific skillsets) makes this route prohibitive to scale.
ThingWorx Studio flips this model on its head by making the content authoring process what it should be—a content authoring process—rather than a coding process, which reduces AR app development time from 1 week to 1 day and uses just one-third of the headcount.
The second challenge is cost. Up to this point, HUD hardware has been prohibitively expensive to procure at enterprise-scale and the cost to develop enterprise applications, usually in the tens of thousands of dollars, has been a fraction of what it takes to create a tailored AR experience. But this is starting to change. Hardware is becoming cheaper as key components (optical modules, depth sensors, SOCs) become more widely available. Software development costs, the primary costs being time and labor, are also dropping because devices are becoming more accessible and vendors are expanding their offerings to include integrations with both new and existing enterprise infrastructure, in addition to traditional EMM/MDM tools.
The third challenge is market maturity. The AR market is still nascent and awareness is low. However, many enterprises that were testing AR solutions in 2015 and 2016 are now beginning to deploy them. This is where AR development platforms can help accelerate
AR adoption by operationalizing not only the deployment process, but also AR content delivery and application management.
While the AR market has been slowly overcoming barriers, the market is rapidly progressing in a way that extends AR experiences to connected things to capture the benefits outlined above. However, there are several challenges in building IoT applications to support
AR application development efforts.
The first challenge is connecting a machine or device to provide the data accessible to AR applications. Connecting a product can first involve sensorizing the machine or thing, which may require engineering expertise if off-the-shelf sensors are not available. This activity can also include use of software agents on the machine, messaging technologies, and networking protocol translation
services. Translation services are often required in industrial markets that use field bus or industrial Ethernet networking protocols.
Physical transport of data from the machine is another requirement provided by multiple technologies, including fixed line, LAN, and
WAN wireless connections and satellite. Once data are in the cloud, the data typically need to be normalized and categorized so applications can identify the data sets to extract from. All technologies and services that extract data from a machine or object and deliver them to a cloud are called application enablement activities, which can also include writing business rules to act on machine data, as well as device management services.
These “plumbing” activities require selecting the right technologies, but even before making technology decisions the second challenge of identifying and selecting the right suppliers arises. The IoT boasts a vast supplier base, particularly for machine connectivity and application enablement services. In these parts of the IoT value chain alone, there are hundreds of suppliers to choose from, which is why supplier diversity and offer complexity are consistently considered top challenges for the IoT market.
Competent IT teams or SI partners can navigate these issues and develop customized AR applications. However, the difficulty is not in a single development effort, but in creating a scalable and sustainable application development capability that expands AR applications to a broader set of connected machines and continuously upgrades AR apps with new IoT data.
Driven by such a complex IoT market, the supplier community has responded by creating IoT platforms that aggregate many of the technologies and services for building IoT applications. The question is can these platforms be used to help build AR applications for connected products. The following are key requirements of an IoT platform for supporting AR applications both today and in the future.
Device Connectivity Services
As noted earlier, before an AR application can enjoy the value of IoT data, the machine or device needs to be connected. This can include sensorization and data digitization services, protocol translation services, knowledge of device OS platforms, and relationships with gateway and communication providers. At a minimum, IoT platforms that offer a full suite of connectivity technologies/services greatly simplifies building a connected asset and speeds overall time to market for advanced AR applications.
IoT Application Development Assets
For AR applications, IoT platforms need to have, at a minimum, business rule development tools that describe actions based on values and patterns detected in the IoT data. These actions, which align to needs of the business or occupational task, can be displayed in the AR application; for example, the number of alarms a motor registered between inspection events. Another less often found application development asset in IoT platforms is composer tools for creating an application from IoT data. These tools can be as extensive as drag and drop environments for fast build and application deployment to a mix of traditional software development tools augmented with IoT-specific drag and drop menus. Composer tools are not necessary for AR applications, but can have a democratizing effect in driving more IoT services into an organization that can then accelerate AR application development.
Digital Twins (CAD Integrated IoT Cloud)
CAD programs, by design, build a digital twin of a device or object and AR applications and can leverage the digital twin to improve data assignment for different object features. When CAD software is enhanced with IoT data, a powerful combination is created for building AR applications. Now, developers do not have to go through the laborious process of integrating both business rule and machine attribute IoT data into the CAD-generated digital twin for use by the AR application. Pre-integration of IoT cloud data into CAD software removes these development tasks and speeds AR application development.
Analytics
Companies that have implemented IoT solutions are quickly moving from basic descriptive analytics to advanced analytics that can predict events such as machine failure, or simulate future events called prescriptive analytics. For AR applications, advanced analytics information supercharges its value. For example, field services personnel using an AR application to replace a seal or filter, may see that recent operating data suggest a more robust filter to extend the machine’s lifetime under new operating conditions. This benefits the customer and the service organization,potentially eliminating an unplanned maintenance trip. IoT platforms that offer advanced analytics services are not common, but they are incredibly valuable for both IoT services and AR applications. Overall, the IoT cloud facilitates development of advanced analytics and makes it available to the AR application.
System Integration
Because IoT platforms enable applications, many offer pre-built connectors from the IoT cloud to common enterprise applications. This speeds time to value as IoT data can now be applied to more applications and, by default, be usedby more employees accessing these applications. For AR applications, with the IoT cloud as the common integration point, IoT-augmented enterprise application data can more easily be delivered and presented in the AR application.
The proceeding IoT platform capabilities greatly simplify building IoT-enhanced AR applications. However, other even more important long-term benefits can be derived from using IoT platforms.
Scalability
IoT platforms provide a scalable way to both expand IoT to new AR applications and to enhance current AR applications with new IoT data. In the former case, new AR applications can leverage the same IoT cloud, whichhas pre-established connectors for machine data and business rules. In the latter case, as analytics on machine data
advances to new levels, predictive and prescriptive insights can be delivered to the AR application.
Agility
Businesses may not know which IoT data to incorporate into the AR application. IoT platforms provide flexibility and agility for selecting IoT data that provide value based on the business process and occupational need. Today,businesses building AR applications will show simple machine operating statistics data, but may later find the need to add product and service recommendations derived from machine data analytics.
Security
Once an asset is connected, security risks are greatly elevated. IoT platforms can reduce the security challenges through pre-approved device components. The approval process requires testing communications with the IoT cloud where security issues and concerns can be addressed. AR applications benefit because IoT data found in the application will be used to interact with the object, which need to be true for operational and safety reasons.
CASE STUDY
Griiip is a startup-level racecar manufacturer based in Israel on an ambitious yet attainable quest to lower the cost of racing by building an affordable product that is also easy to maintain. The company’s first car, the G1, is a Formula 1000-spec, single-seater that weighs less than 1,000 lbs and can achieve top speeds in excess of 150 mph thanks to its 187 HP engine and limited slip differential (no chain, zero maintenance, high performance). But this is not an easy business to get into; racecars are expensive, as is the engineering team needed tosupport both training and race day activities. These costs can amount to more than US$200,000 for a single season (excluding the cost of the car, because the price tag for a traditional Formula 1 vehicle is such that drivers often pay, typically thousands of dollars, for access to a car/team, rather than purchase it. Griiip’s goal is to allow individuals the opportunity to buy the car, not the racing team, at half the cost. All of these factors supported Griiip’s desire to find a comprehensive yet consolidated set of tools that could quickly move data between programs/backend systems through a common platform.
Griiip took a very forward-looking view to meet its objectives. The challenge was that it not only wanted to build a compelling product at an affordable price, but one that could create incremental value for the company, as well as the drivers, fans, and followers. After examining the various solution enablement options, Griiip decided to enroll in PTC’s 4-month ThingWorx Studio Pilot Program, which gave it early access to a suite of industrial AR tools that could be integrated with its IoT application enablement platform (ThingWorx) and leverage existing 3D CAD data (e.g., from Creo) to accelerate the content authoring process, without the need for additional coding. These integrations allow Griiip to monitor the health of its G1 vehicles in the field, provide visual and virtual coaching-style cues to drivers (e.g., to improve performance), and offer fans and followers the ability to view live data on drivers during a race.
Data collected by the G1’s dozen-plus sensors enable Griiip to remotely monitor and analyze the condition of its vehicles. Aside from race-related driver tracking, this information can be used to notify a G1 owner if and when maintenance is needed, and identify the appropriate course of action for a repair, which is typically to hire a mechanic. This is one of the key reasons why ThingWorx Studio-enabled AR experiences have become a centerpiece of Griiip’s go-to-market strategy; it allows drivers to act as the expert through a combination of 3D product navigation and step-by-step instructions (viewable via smartphone, tablet, or HUD), and takes the need for third-party support out of the racing picture.
For Griiip, which intends to manufacture several thousand G1s and will see its first series of vehicles hit the track in 2017, this level of visibility, and knowing the state of every car, all the time, makes it much easier to manage spare part manufacturing, improve product development, and promote the customer experience with better service, at a lower cost.
CONCLUSIONS
AR applications will drive the next wave of enterprise productivity through the delivery of more contextual and operationally relevant data overlaid onto a live image of a machine or object. Field services and maintenance workers will benefit the most, particularly in
manufacturing, mining, logistics and supply chain, oil and gas, utilities, and construction. Interestingly, these same industries are also aggressively pursuing use of IoT technologies and services to improve maintainability and reliability of the equipment and machines driving their businesses.
IoT platforms greatly accelerate the benefits of IoT applications and services that monitor equipment, predict machine outcomes, and develop better products. IoT platforms simplify the selection of technologies and suppliers for machine data collection and integration into other IoT applications. Advanced IoT platforms offer deep integrations into other enterprise systems, including CAD, PLM, SLM, and ERP, as well as access to new predictive analytics tool sets.
While AR applications are valuable without IoT data, when IoT data are added, AR applications become supercharged and IoT platforms become a powerful AR application enabler. But the benefits of IoT platforms for AR application go beyond enablement. IoT platforms provide a scalable, agile, and secure way to cost-effectively enhance AR applications and extend both AR and IoT services across the business.
