Home Press Coverage

Remote Machinery Monitoring – a Developing Industry Nelson Baxter, Mooresville, Indiana Heather De Jesús, Azima, Inc., Woburn, Massachusetts Remote monitoring of rotating equipment has been around in one form or another for several years. With the advent of wireless communications and the Internet, it is now possible to wirelessly communicate with machines and view data via web site interfaces. These changes in technology allow data to be analyzed remotely by experts so that actionable advice can be supplied to the end users who are operating the equipment. Two of the most common words used to describe our economy today are “high technology” and “service.” Wireless-web-based remote monitoring by off-site specialists is a prime example of the unification of these two concepts. Monitoring the health of rotating equipment has come into and out of favor several times over past decades. Lapses in monitoring created by job cuts or reorganizations are followed by unanticipated failures, which then reignite interest in predictive maintenance technologies. The reason for this type of pattern is that predictive maintenance is an important – but noncritical – function. What this means is that if you do not perform predictive maintenance, then nothing happens right away. Like not going to the dentist, which is also an important but noncritical function, after a long enough time, it suddenly becomes very critical. Installing remote monitoring systems helps narrow the swing of the predictive maintenance pendulum by decoupling the predictive maintenance program from the swings that are tied to the availability of in-house personnel. Some of the driving forces that help promote a consistent predictive maintenance program are presented below. Predictive maintenance programs make sense. From an economic perspective, it is logical to replace a $500 bearing rather than to repair $5,000 in consequential damages or have a $50,000 loss in downtime. Since by its very nature predictive maintenance prevents these failures from occurring in the first place, it has always been difficult to get a handle on the benefits. It becomes a little bit like buying life insurance. It is sometimes hard to justify, but a wise manager does not want to gamble and do without it. Certain quality programs like ISO certification require facilities to include predictive maintenance on machines that are critical to the process. Regulatory agencies like the Nuclear Regulatory Commission (NRC) or the Food and Drug Administration (FDA) require testing of the condition of equipment that is critical to the safe operation of the plant or vital to the quality of the production process. These factors that produce a need for a consistent program have been present for a long time. What has changed recently is availability of technology that makes it practical to do so from the perspective of remote monitoring. A Paradigm Shift in the Future It is hard to predict the future precisely. As well known futurists have indicated, there are some things we can do to at least get an idea of where we are headed. For instance, you cannot predict what will happen tomorrow. But you can predict with a fair degree of certainty, based upon past experience, that the earth will keep turning, the stock market will go up and down and there will be trouble somewhere in the world. We can therefore start to look into the future by studying the past and then looking at the forces of change and our available options in regard to responding to these forces. One well known futurist gave the following thoughts regarding agriculture and manufacturing. “If we look at past stages of development, we can see, in the case of agriculture, that the system became more and more efficient. We still grow incredible quantities of corn, wheat soybeans and cotton; we just do it with a small fraction of the personnel that it used to take. The same is true with mining and manufacturing. Tremendous amounts of coal are mined and vast quantities of goods are produced, it just takes far less people to do the work.” Based on what has happened to other industries, we can predict with a pretty good degree of certainty that the information industry, of which predictive monitoring is a part, will have to get more done with less people. In the predictive maintenance arena, part of the efficiency increase will be attained by getting smarter on how we monitor things. For instance, should all equipment in a plant be monitored on a monthly basis as is often done? Most likely that is not the case. Should very critical equipment be monitored more frequently? This is also probably true. We can therefore increase both effectiveness and efficiency by more closely studying which equipment to monitor and how often to do so. The greatest potential for efficiency improvements in predictive maintenance, however, lies in the synergism of combining the technologies that are either presently available or in development. Wireless sensors coupled with in-plant networks and accessed by the Internet promise to automate the collection and logging of data. These data will then be available for viewing by analysts anywhere there is an Internet connection. The vibration analyst could literally be sitting in an airport, at home or in a motel with a wireless connection looking at the condition of his critical machines and sending correspondence to operations on whether or not the equipment will make it to the next outage. What will predictive monitoring look like in the future? Based on present economic forces, the past history of other industries and the technology already available or shortly to come on line, we can describe a number of scenarios. There will be fewer personnel collecting data and performing analysis. The combined economic forces of global competition, skyrocketing health care costs, deregulation and demographic shifts are so compelling that this is nearly a certainty. Data will increasingly be brought to the analyst rather than the analyst going to get the data. Note that this will almost surely be true for periodic data collection, first-stage problem detection and basic analysis. However, when a problem is complex, it will still require, militarily speaking, “boots on the ground.” An editorial from a few years ago titled “The Perils of Troubleshooting in Cyberspace,” goes into the details of why solving machinery problems while at a computer screen can be difficult. Detection and basic analysis can be performed remotely, but specialists will still have to go on site to solve certain classes of problems. Data from similar machines will be archived so the wheel will not have to be reinvented each time a problem develops. The nuclear industry will probably be the first to use this approach. If there are 37 nuclear units with the same design of containment spray pumps or residual heat removal pumps, then there are obvious advantages to maintaining a common database. After nuclear plants, there will be small process plants that have numerous sites with identical equipment scattered throughout the world. The combination of wireless in-plant data transmission, plant networks and the Internet will become a powerful force enabling the collection and transport of data to the first-line analyst and then on to specialists, if necessary. The problem to date has been what the phone people call the dreaded last mile. This refers to the fact that you can set up the trunk and branch lines of a fiber optic network, but the difficulty lies in the need to go the last mile into the millions of homes and businesses. The thing that will solve the last-mile issue (the numerous connections to the bearings on each machine) in regard to predictive maintenance may be a wireless connection between transducer hubs on the machines and the plant network or an Internet connection. The approach to predictive maintenance will vary from plant to plant. In large plants with hundreds of machines, for instance, there will probably be a hybrid solution that consists of machines vital to the operation being wirelessly monitored on a daily basis and the nonvital equipment being manually monitored maybe every six months. This would reduce the amount of manual monitoring to a small percentage of what is being done today and at the same time provide better coverage for the vital equipment. As costs come down, the number of machines that can be monitored wirelessly will increase in these large facilities. Intermediate-size plants that cannot justify on-site personnel will need to choose between contracting out a manual service and going to a contracted wireless service. Small plants, like those that supply liquid oxygen, do specialty chemical processes or perform pumping operations that are geographically scattered throughout the country would best be served by entirely going to remote monitoring to reduce travel time required to go from site to site doing manual monitoring. Since these types of plants use nearly identical equipment at dozens of facilities, they will be the ideal situation for remote monitoring, with all the data going to a common point so a knowledge base can be acquired and the data mined for use in fine tuning future problem prediction. The key to survival of the analyst and to the companies they work for is to recognize what is coming and be on the forefront of making it happen rather than lamenting the past. Several companies have recognized the paradigm shift away from manual monitoring to remote. The following sections describe the advantages of remote monitoring and the methods being used. Ten Reasons to Use Remote Monitoring Remote monitoring leverages available personnel by allowing them to spend more time looking at the condition of the equipment and less collecting data. Reductions in personnel have put a strain on the predictive maintenance organizations of most companies. Layoffs, restructuring, early retirements and mergers have devastated what were formally well-functioning predictive maintenance organizations. In addition to reducing the number of personnel, the training departments that trained these personnel to begin with have also been reduced. The people that are left need to be more efficient in their jobs. Certain types of industries do not allow personnel into critical areas that contain vital equipment. If they are allowed to enter, then special procedures or personnel protective equipment are required. Nuclear power facilities, for example, require dosimitry, special training and protective clothing to go into certain areas. Pharmaceutical production facilities have areas that, once they are sealed, do not allow anyone to enter. Chip and nanotechnology production facilities do not allow entry to ensure that there is no product contamination. Chemical factories have areas that are restricted due to hazardous chemicals. Once installed, remote monitoring systems can make the machinery health data available anywhere in the world. There are also safety considerations. Large cranes, conveyors, drag lines, open drive shafts and open gear sets all pose dangers to personnel collecting data via normal walk-around methods. With a remote monitoring system, data can be obtained day, night, weekends, holidays and when personnel are sick, on vacation or away for training. Even more importantly, the program continues to function during periods of personnel turnover. Remote systems can be set up to easily collect other types of data that might be important to performing a better analysis. Examples are current flow, speed, pressures, temperatures and flow rates. The combination of the above parameters gives a much better picture of the overall health of a piece of equipment than does vibration alone. With these additional data, efficiency calculations can be made in concert with vibration analysis. When manual routes are run, inevitably some machines will not be in operation. This can result in surveys on some machines being missed for several months at a time. For any given number of machines, there will usually be one or more that is having a problem. With a remote system, the rate at which data are taken can easily be ramped up to watch the machines in trouble more closely until the next scheduled downtime. It is often difficult to have someone with a manual system go out to a problem machine several times a day, or at night or on weekends. With remote monitoring systems watching similar machines at different facilities, it is possible to bring all the data into a common database. This makes it possible to perform a better analysis. In addition, the remote data can easily be viewed by other analysts or equipment manufacturers, making it possible to network multiple analysts and equipment manufacturers. One of the major advantages of using remote data collection is in setting alarms. A remote system can be ramped up to collect a large number of data points that can then be used to set up accurate alarms. If data are collected manually once per month, then the number of data points would probably be 10 or 11 per year. A remote system can be programmed to take data frequently, and several hundred readings can be statistically analyzed to set up alarms. Data from supervisory panels mounted on vital machines can be monitored remotely. There is a much higher likelihood that a machine trip due to high vibration will be caught with a remote monitoring system than with a walk-around program. Remote Monitoring Philosophies Remote monitoring methods can vary from simple to complex. The simplest form of monitoring could be taking the output from a vibration transducer, converting the signal to a 4-20 mA or DC value and then sending that value to a plant data logging system that can be viewed via an off-site connection with something like Remote Desktop. Obviously, very little information is gained other than the vibration being high, normal or low. Because there is no spectral information, the cause of an elevated level cannot be determined. The output from a real-time analysis system that interfaces with a site server accessible with a VPN (Virtual Private Network) connection would represent the other extreme. Such a system might offer spectra, waveform overall trends, DC gaps and orbits. The most complex systems are able to track transient conditions such as startups and coast-downs. They can produce Bodé and polar plots that show the location and severity of critical speeds. They can also perform advanced functions like run-out subtraction. Two questions a customer must ask: How much does a system cost? Who is going to look at the data, interpret what it means and provide advice? Spending a lot of money, only to be overwhelmed by gigabytes of data that no one looks at or knows how to interpret, is a futile waste of time and resources. All too often, data efficiently flows into a data storage black hole where it is never viewed, much less interpreted, by knowledgeable people. A medical analogy may be useful here. An experienced doctor can measure your pulse, take your temperature, ask you a few questions and make a pretty good diagnosis of what is most likely wrong with you. Give him some more information and he will do even better. On the other hand, if you were to undergo blood chemistry work, CAT scans and EKGs, if the person looking at the data is not trained, then it is a waste of time and resources to supply them with all this complex data. The important point to make here is that when the decision is made to perform remote monitoring, a key element in deciding what level of monitoring to choose is: who will be looking at the data? It makes little sense to install an elaborate transient analysis system if no one is qualified to examine the information. On the other hand, if qualified personnel are available, then it would not make any sense to restrain their ability to analyze problems with only overall levels that have no associated spectra or waveforms. It becomes apparent that if we are to obtain advanced data, then qualified people are needed to interpret provided information. This is where the concept of remote monitoring becomes important. With the availability of data over the Internet, it is now feasible to connect the data with those qualified to interpret it. These specialists can make recommendations to maintenance and operations personnel. Therefore, it clearly makes sense to obtain reasonably advanced information, since it can now be economically used. What approach should be taken then with regard to the quantity and technical complexity of data obtained with a remote monitoring system? As with most things, a middle-of-the-road approach probably makes the most sense. Supplying overall values is inadequate, because it just lets the analyst know there is a problem. Overall values provide almost no useful information as to what the problem might be, let alone what maintenance or operations actions should be taken. On the other hand, installing very expensive systems that monitor startups and coast-downs and collect and store megabytes of data each day are probably overkill. The middle of the road would be to take overall levels, spectra and waveforms on a periodic basis. Phase, DC-gap voltages and orbits could be taken when proximity probe systems are available. Current, speed, pressures and temperatures, which are scalar values, would also be desirable parameters to monitor. If the amplitude of the overall values increases, then two things need to occur. First, the rate of data collection needs to increase, and second, the doctor needs to receive notification that the patient needs to be examined. The Internet is the perfect means to both notify the doctor and provide access to the data. The Internet is the enabling technology that allows the machinery expert to view the vibration data and make requests for the monitoring system to take more detailed information. Examples might be to temporarily increase the rate of data collection, increase the spectral resolution, or alter the maximum frequency range. The amount of this that might be set to occur automatically might need to be adjustable to achieve proper balance between being overloaded with data on an alarm and having adequate information with which to perform an analysis. The key features are automatic notification, ability to look at the data from anywhere and the option to change analysis parameters remotely. Methods of Remotely Viewing Data The following are some ways to view data from a remote site. Something as simple as using the Windows Remote Desktop application can allow an analyst to view data at a remote site. Another possibility is to have an on-site server that can be accessed by a virtual private network (VPN) connection. The server contains the data from the data collection system. To access the data and manipulate the displays and have cursor functionality, special client software is usually required on the computer that is accessing the data on the site server. The third and most flexible approach is to have the data sent to a remote computer where it can be viewed by a standard browser like Internet Explorer or Firefox. This third approach has several advantages and will likely become the choice in most future remote monitoring applications. One major advantage is that anyone who has the user name and password can view the data without installing client software. It is just like logging onto a web-based bank account, but instead of seeing banking information, the user sees trends, spectra and waveforms. The web-based approach makes it possible for customers, analysts and equipment manufacturers to all simultaneously look at the data from a piece of equipment, collaborate and make a decision. All this can be done on any computer from anywhere in the world without any special software. One of the main advantages of the web-based approach is that most people already know how to navigate a web site using a mouse and all the normal browser features. This means that, with a web-based system, it takes little or no special training for a user to be able to interact with the data. Acquiring the Data There are different approaches to transferring data from a machine in the middle of a facility to a computer where it can be viewed by an analyst via a remote connection. If the Internet is the trunk of the tree, then the roots represent all the multiple connections that have to be made with the individual bearings on each machine. There are various methods for getting the data from the machines being monitored. The old method would be to run cables from the transducers to a data acquisition unit that transfers the data to an on-site computer. This can be very costly. If there are 100 transducers scattered around a plant that is the size of a city block, then the cost of cable alone can make remote monitoring uneconomical. For instance, if the fans in a power plant are 500 ft from the control room and there are 10 transducers on a fan, this means that it would be necessary to run nearly a mile of cables through conduit and cable trays to monitor one fan. A second approach would be to run sensors to a local hub and connect that hub to the central server by a single Cat 5 Ethernet cable. This approach significantly cuts down cabling costs. Since the data are digitized at the hub, it reduces the chances of interference compared with running analog signals in cable trays. A third option and one that is becoming increasingly attractive is to transmit the data wirelessly to the on-site server or to some systems directly off site. If wireless access points are already available, then this approach eliminates the wiring completely. There are two methods of approaching wireless transmission to access points. The first is to use a battery-powered, combined sensor-transmitter that sits on the bearing. Two things to consider with a combined unit are that the bearing may be hot enough to damage the electronics or where the bearing is located may not be the ideal spot for wireless transmission. Another consideration is if a combination sensor-transmitter unit fails, then the whole thing may need to be replaced. This could be costly. A second approach is to use standard sensors with short wires that go to a local sensor hub that samples the data and transmits it. These hubs can be located away from heat and leaking seals and elevated above the machine for better data transmission. Failure of a transducer is not a major problem, since they are standard shelf items. This type of sensor hub might be either battery or AC powered. Battery-powered units are acceptable when data are only taken once or twice a day, but if more frequent monitoring is needed, then line power would be needed. Wireless data transmission is available in three main forms: 2.4 Gigahertz, 900 Megahertz and Bluetooth. The latter has limited range, so it is not utilized much in an industrial setting as compared with the two previous choices. The 900-MHz format offers the best distance and penetration ability but has been found to cause interference in some plant environments, and its uses are not standardized. Of the three, the 2.4-GHz, spread-spectrum technology has proven to be the most effective in a plant environment. A study funded by utilities showed that it does not interfere with plant controls and can be used in multiple applications. If a plant installs a system of 2.4-GHz access points, then it can be used for numerous other applications. In-plant voice over IP (VOIP), in-plant computing, hand-held PDA-based data loggers, dosimetry and vibration systems can all share a single 2.4-GHz system simultaneously. The list grows every year. At one facility, a test was run with laptop computers, cameras and several vibration sensor hubs all operating simultaneously with no problem on a 2.4-GHz wireless system. Network Security Issues One of the most difficult issues that has to be faced when setting up a remote monitoring system is network security. Having a wireless network that might be hacked is one concern. Sending commands to sensor hubs behind a firewall is something that concerns those responsible for plant network security. Another potential concern is having data on key machines flow out from a facility. Some customers might be concerned that data from their machines might be intercepted and used by a competitor. For instance, a utility might worry that if someone knew that they had a turbine that was in trouble, the information might be used for economic advantage by a competitor. The 802.11b wireless protocol supports several types of wireless security encryption. Wireless encryption could be thought of as ‘passwords’ to be allowed to communicate on an encrypted wireless network. The different types of encryption are wired equivalent privacy (WEP) and Wi-Fi protected access (WPA/WPA2). WEP encryption has two versions – there is 64-bit and 128-bit encryption. The difference between 64 and 128 bit is the length of the encryption key (a.k.a. password). Wireless networks using 128-bit encryption often have not just one but four different ‘passwords’ referred to as encryption keys. WPA/WPA2 encryption use the same basic concepts as WEP but are considered to be more secure, since they are the newest encryption protocol. In addition to the encryption protocols available with wireless networking, there are also different types of authentication. Authentication is how wireless access points initiate communication with a client. There are two ways of initiating communication – open key and shared key. Open key allows anyone to start a conversation with the access point (AP). If the wireless client initiating the conversation does not have the correct WEP, WPA or WPA2 key settings, then the AP will not allow that client to further communicate. Shared key requires that the wireless client initiate communication by specifically requiring that the security information (keys) be passed first. It basically works as follows: 1. A client begins by sending an association request (request to communicate with the AP); 2. The AP responds to the request with a “challenge text” (unencrypted) to the client; 3. The client using the proper WEP key encrypts the challenge text and sends it back to the AP; and 4. If the client properly encrypted the challenge text with the proper encryption keys, the AP then allows the client to communicate. There is an additional level of security available on the AP to help ensure no unauthorized wireless clients gain access to the wireless network or media access control filtering. A media access control (MAC) address is a unique code assigned to most forms of networking hardware – almost like a Social Security Number. With most APs, one would have the ability to restrict communication to only certain MAC addresses. This in combination with WEP, WPA or WPA2 increases the overall security of the wireless network. Once the wireless network is installed and the level and types of authentication and security protocol are chosen, the wireless network could be segmented for different types of data traffic. For instance, one may want to separate plant control systems from corporate business data traffic. This could be done several ways, but the most common and cost-effective way is to use a tool available on more sophisticated APs referred to as ‘truncation.’ Truncation allows an AP to set up several different virtual localarea networks (VLANs) all on the same AP. VLANs help segment different types of traffic (plant control from corporate business and/or plant conditional monitoring systems). Wireless clients would only be allowed to communicate and pass data on the specific VLAN to which they were configured unless a particular client was granted specific access to the another VLAN within a given AP. This access would be granted on the AP and would need to be set in the AP’s configuration file. A final step that might be used in ensuring data integrity is to set up a virtual private network (VPN), which provides remote access to an organization’s network via the Internet. VPNs send data over the public Internet through secure ‘tunnels.’ Remote Monitoring Case Histories Case 1. A large fan in a steel mill was being watched with a remote monitoring system. The trend is shown in Figure 1. The plant tried some new grease on one of the motor bearings. The plot shows that in the 4 hours following the addition of new grease that the vibration went up 400%. Had data been taken once per month, the connection between the new grease and the increase in vibration would have been missed. In this case, the analyst received an e-mail message identifying the alarm. The analyst immediately called the plant and the connection to the greasing operation was made. Case 2. A forced-draft fan at a utility had a bearing failure that was detected by a remote monitoring system taking data once per hour. The bearing was replaced prior to any consequential damages or downtime occurring. Figure 2 shows that the impacts reached a level of 10 g in a relatively short time. Case 3. A 10,000-HP motor driving a compressor at a liquid-oxygen plant was sent out for overhaul. The motor had been monitored with a remote system. After overhaul, the vibration had increased. Since data had been taken right up to the day that the machine was removed and immediately following the reinstallation, it was possible to positively tie the increase to the overhaul activity. The motor shop had not balanced one of the rotating components properly. When shown the data (Figure 3) from the remote monitor, the motor shop agreed to correct the problem with field balancing. Case 4. A circulating water pump at a nuclear power plant was on a remote monitoring system. Following an outage, the vibration amplitude was discovered to be much higher than prior to the outage (see Figure 4). Since it was the hottest part of the year, the pump was needed badly. The remote system was set to check the data every 2 hours and notify operators at a preset level. The remote system saved the plant personnel a significant amount of time by automatically monitoring the pump until it could be repaired. Case 5. A large through-air dryer fan at a paper mill had higher- than-normal levels of vibration. The fan was brought off line, and it was discovered that there were cracks in the wheel. The plant wanted to operate the fan for another six weeks until the next scheduled outage. For safety reasons, a remote monitoring system was installed. When the process was brought down immediately following the down period, the vibration would spike (Figure 5). It was discovered by studying the spectra that when the process was restarted, the automatic control system was over-speeding the fan. Considering that the fan already had cracks, this over-speeding was both dangerous and increased the likelihood that the fan would not make it to the outage. The operators were informed of the problem and they then changed to manual operation. This allowed the fan to make it to the scheduled outage without failing. It would have been extremely unlikely to have caught the over-speed problem with a manual system. The fact that qualified personnel were watching the fan remotely and recognized the increase in speed was equally as important as the remote monitoring itself. Case 6. A problem was detected with a bearing on a large wetend exhaust fan at a paper mill. A temporary remote monitoring system was installed, with spectra and waveform being taken every 10 minutes. Based on the increase in the trend level and the high level of impacts (see Figure 6 and Figure 7), the analyst recommended reducing the fan’s speed for it to survive until the next scheduled outage. That action was taken and the fan survived. When the bearing was examined, all the rollers were found to have extensive damage. Case 7. A Hoffman blower had a bearing that was failing. Figure 8 shows that the blower hit the first alarm on Christmas and the red alarm on New Year’s Eve. The remote system watched the progression and sent out e-mail messages on both occasions. The bearing was replaced shortly after. This example shows another advantage of remote monitoring in that data are taken day, night weekends and holidays. Conclusions The world of predictive maintenance is changing. Global competition, mergers and early retirements are causing reductions in the number of personnel who can be dedicated to monitoring equipment. Safety and process requirements make it difficult to monitor certain classes of equipment. The combination of wireless technology and Internet access to data now enable fewer personnel to safely monitor equipment from anywhere in the world at any time day or night. The authors can be reached at nbaxter@true802.com, hdejesus@azimainc.com.