US20070025528A1

US20070025528A1 - System and method for automated performance monitoring for a call servicing system

Info

Publication number: US20070025528A1
Application number: US11/176,972
Authority: US
Inventors: Benjamin Knott; Robert Bushey; John Martin
Original assignee: SBC Knowledge Ventures LP
Current assignee: AT&T Intellectual Property I LP
Priority date: 2005-07-07
Filing date: 2005-07-07
Publication date: 2007-02-01
Also published as: US8175253B2

Abstract

A system and method for monitoring and reporting automated call-processing performance is disclosed. Initially, performance metrics can be established for normal or acceptable system performance. Performance metrics may be determined for individual components and/or a combination of components in a system, wherein the individual metric may help to provide a performance indicator for the call processing system. The performance metrics can be established by utilizing historical data gathered from past operating characteristics or operating parameters. The performance metric can be calculated from past operating characteristics and could be a statistical distribution or a statistical value. This statistical distribution will often conform to a Gaussian distribution also known as a bell curve. However, acceptable performance limits can be set based on nearly any statistical distribution. A monitor can collect data over a predetermined time interval and process the monitored data to provide a “near” real-time performance data. When the near real time performance data falls within a pre-defined range of normal operation (based on the statistical distribution) system operation is most likely acceptable and operation proceeds. However, when the system performance data is outside of a normal range of operation, beyond a performance metric or exceeds a threshold value an alarm can be set to notify a human or an electronic device.

Description

BACKGROUND

Telephone call servicing centers that have interactive voice response systems (IVRS)(s) continue to improve by implementing new technologies. One such recent development is a speech recognition technology that supports a “natural language” type interaction between automated systems and customers. A natural language interaction allows a customer to speak naturally, and a voice recognition system can route a call or provide requested information responsive to the freely spoken dialect. A goal of an IVRS is to determine why a customer is calling a service center and to provide the customer with information, or route the customer to an appropriate agent or destination. Ensuring that a given IVRS is operating properly and accurately is often very difficult. An IVRS is typically comprised of multiple components that can operate at less than peak efficiency or even fail. One significant problem that occurs in IVRSs is that certain failures degrade system performance but do not cause total failure, wherein detection of such a failure is difficult. Call centers often service a huge volume of calls and even if a small percentage of calls are improperly serviced, the costs associated with such calls are significant. In addition, unsuccessful call servicing can result in caller dissatisfaction. Accordingly, there is a need for an improved method and system for tracking automated call center performance.

BRIEF DESCRIPTION OF THE DRAWINGS

It will be appreciated that for simplicity and clarity of illustration, elements illustrated in the Figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements are exaggerated relative to other elements. Embodiments incorporating teachings of the present disclosure are shown and described with respect to the drawings presented herein, in which:
FIG. 1 is a block diagram of an exemplary telecommunication system having automated call processing features;
FIG. 2 is a flow diagram that illustrates a possible method of operation of the system of FIG. 1; and
FIG. 3 is a graph illustrating exemplary “bell” curves.

DETAILED DESCRIPTION OF THE DRAWINGS

A system and method for monitoring and reporting automated call-processing performance is disclosed. Initially, performance metrics can be established for normal or acceptable system performance. Performance metrics may be determined for individual components and/or a combination of components, wherein the individual metric may help to provide a performance indicator for the call processing system. The performance metrics can be established by utilizing historical data gathered from past operating characteristics or operating parameters. The performance metric can be calculated from past operating characteristics and could be a statistical distribution or a statistical value. The statistical distribution will often conform to a Gaussian distribution also known as a bell curve and acceptable performance limits can be set based on the Gaussian distribution.
A monitor can collect data over a predetermined time interval and process the monitored data to provide a “near” real-time performance data. When the near real time performance data falls within a pre-defined range of normal operation or under a threshold value, system operation is most likely acceptable. However, when the system performance data is outside a performance metric or exceeds a threshold value an alarm can be set.
The method can utilize historical activity to determine an “average” functionality and normal operational variance around the average. Recent events can be compared to these parameters to evaluate the acceptability of system performance in near real-time. The time sample utilized to average the system activity and caller behavior can be adjusted to “filter” data resulting from known abnormal events such that they do not skew the data distribution. The triggered alarm can activate an e-mail message, a pager, a phone message, and/or some other communication to a human administrator of the system reducing response time for an underperforming system.
A mean and a measure of variance (e.g., standard deviation) can be calculated from a collection of historical data that is indicative of system performance. These metrics can be calculated over any meaningful time interval depending on the needs of the business. The time interval may be referred to as the temporal resolution of the system. In addition, an administrator of the system may select the threshold level at which an alarm will be activated, based on the desired system sensitivity. In one embodiment the time duration for the collected near real time data can be set to a half hour and the threshold value may be set to plus or minus 1 standard deviation from the mean. In this case the temporal resolution is one half hour and the sensitivity to operational variance is equal to one (1) standard deviation.
Some exemplary operational characteristics that can be monitored can include an average call duration, a system response time, a premature disconnect count, a no caller input count, a voice extensible mark up language (VXML) server access time, a caller hang up rate, an excessive confirmation rate, a counter for all parameters, and a false acceptance rate. The system for monitoring performance of the call center described herein can also be implemented in a touchtone response system or any other electronic servicing system that accept human input or interfaces a computer with humans.
In one embodiment, the system can include a server coupled to the IVRS and configured to supply executable objects to the IVRS. A monitor may also be coupled to the IVRS and configured to monitor at least one parameter or performance characteristic of the IVRS related to system operation or caller activity. Statistical analysis on the performance characteristic can provide a performance indicator that is related to system operation. The monitor can also maintain a performance threshold and determine if one or more statistical operating parameters has deviated sufficiently from the performance threshold to initiate an alarm.
A typical speech recognition system goes through an iterative tuning phase, in which the system adapts to different inputs. Often, changes are made to the user interface and the speech recognizer. During this phase and throughout the lifecycle of a speech application, it is very desirable to objectively measure how the system is performing. Thus, as the system tunes and/or receives upgrades it would be desirable to measure system performance. Power outages, software updates, changes to the system architecture, replacement of components or changes to the user interface can all have unintended consequences on system performance. Less than optimum performance can result in a bad experience for the caller. If the system is not constantly monitored, a call center can marginally operate while no one is aware that something is wrong or has gone wrong.
Referring to FIG. 1, an exemplary monitoring system coupled to an exemplary and simplified call servicing system (call center) 100 is illustrated. When a caller 102 contacts the call center 100, the call may be processed by a switch 116. The switch 116 can route the caller to an interactive voice response (IVR) system 106 or to a human agent 104. The IVR system 106 can prompt the caller for identifying information and for the purpose of the call.
Depending upon implementation detail, caller authorization and caller data may be retrieved from database/memory 122. During interaction with the caller 102, the IVR 106 may also access voice extensible mark-up language (VXML) server 110 for executable instructions.
During operation, monitor 108 may be monitoring the activities at switch 116, IVR 106, VXML server 110 and/or other devices (not shown) within a call center 100. In one configuration, timer 118 can be resident with monitor 108 to timestamp data and log durations or time periods for activities of components of the call center 100. Particularly, operational data from the switch 116, the IVR 106 and the VXML server 110 can provide valuable insight into operational performance. Database 120 can store the monitored parameters, the time stamped data, the duration of processes, and other operating parameters and characteristics of the system 100.
In one embodiment, system performance data for each metric of interest is stored in database 120. The statistical processor 114 can compute a mean and variance for each of the system metrics of interest. The statistical processor 114 can then convert the data into z-scores. A z-score is a variable (e.g. one of the metrics of interest) that has been normalized such that the mean is zero and the unit of measure (or system setting) is standard deviations (i.e., a z-score of 1 is equal to 1 standard deviation from the mean). The system threshold can then be set to a particular z-score. For example, a performance metric of one (1) would mean that a system performance that is greater than 1 standard deviation from mean performance level is considered aberrant, and an alarm is issued responsive to such a condition.
A Gaussian distribution or a “bell” curve (named after its shape) is illustrated in FIG. 3 to aid in understanding the present disclosure. The curve pinnacle 308 shows the mean operating point at the crest or peak of the first curve 302 and the horizontal lines 304 and 306 show possible threshold values that can set as a performance metric. Performance data that falls between threshold values (the threshold values are depicted as horizontal lines) is considered within an acceptable operational range.
Performance that falls outside of the horizontal lines is considered a low probability event, and thus it is considered likely that this is caused by aberrant system behavior. Thus, the bell curve illustrates a concept of how normal and abnormal system behavior may be defined, where acceptable performance measurements (performance indicators) that fall within the middle of the curve are measurements that occur often, resulting from normal system operation or behavior, whereas, measurements falling within the “tails” of the curve occur less often and with a lower probability that the center of the curve and likely represent a problem with the system. FIG. 3 will be described in more detail below.
Referring back to FIG. 1, and in accordance with one embodiment of the present disclosure a value or measurement for an operational characteristic or system parameter can be averaged over a period of time to provide statistical data regarding activities of selected subsystems. For example, over a ten minute time period an average or other statistical value can be calculated to determine a “near” real-time performance of the IVR 106. The near real-time performance can then be compared to the selected threshold value to determine if the system 100 is providing acceptable performance.
Statistical processor 114 can calculate different statistics for different operational parameters. When a recent statistical calculation provides an operating performance value or a data point that is equal to, or exceeds a performance metric or threshold value, or has an excessive deviation from a nominal performance, an alarm 112 can be activated by the statistical processor 114 or by monitor 108.
In another embodiment, an alarm can be triggered after detection of an extreme value representing an operational parameter. Extreme values may result from a power outage or a component of the system “locking up.” Thus, no time sample may be required for detection of abnormal system performance.
The activated alarm can produce an e-mail, a page, or a phone call intended for an administrator of the system (not shown). The statistical processor 114 can determine an operational performance characteristic over the selected time period and compare it to a multiple of the standard deviation from the mean. In one configuration, the sample time can be set at a half hour and the threshold value may be set at plus or minus 1 standard deviation from the mean. However, the threshold can be adjusted for each operational parameter to avoid alarms that do not accurately indicate an underperforming system.
Referring to FIG. 2, an exemplary method that could be utilized by the system of FIG. 1 is illustrated. At step 202 operational parameters of a call processing system are monitored and data indicative of the operational parameters is stored. From this stored data, a satisfactory performance limit or performance metric is determined at step 204. The satisfactory performance metric can be a high or a low threshold value, a deviation from a normal operating point or a range of values. System operating parameters are monitored and stored at step 206. At step 208 calculations to provide statistics on past system performance may be made utilizing the stored data. Calculations would typically be made for a single operational characteristic over a given time period, however, more than one characteristic for smaller or larger time durations may also be utilized to provide an indicator of system performance.
At step 210, the calculated performance value can be compared to the performance metric or threshold value indicative of acceptable system performance. At step 212, a system may detect when operating characteristics deviate from satisfactory system performance limit or a performance falls below a baseline, a threshold or outside of the satisfactory performance limit. Responsive to the detection, an alarm can be set to notify a user as illustrated in step 214. Generally, when the system is operating within the satisfactory performance limit, it can be assumed that the system does not need user intervention. When call center performance falls below or outside an acceptable level, all or some identified portion of the calls may be routed to a human operator to avoid or minimize caller interaction with a “broken” IVRS.
Referring to FIG. 3 statistical curve, graph, or set of averaged operational parameters may provide a statistical distribution embodied by a curve such as first curve 302. The peak 308 of the curve 302 may illustrate an “arithmetic mean” or typical operating point for an operational parameter. The pinnacle 308 is a metric of an operating point that the system operates at more often than other operating points hence the greater value on the graph. Operating points to the left of the pinnacle 308 may be a better performance and operating points to the right of the curve may be a worse performance or vise versa.
The curve 302 can be representative of any monitored parameter or combination of operating parameter such as an average call duration, a system or component response time, a premature disconnect count, a voice extensible mark up language (VXML) server access time, a caller hang up rate, an excessive confirmation rate, and a false acceptance rate.
First threshold 304, second threshold 306, third threshold 310 and fourth threshold 312 are exemplary threshold values, which may be defined in terms of z-scores or standard deviations from the mean. Thus, a system designer can select a threshold by selecting a standard deviation from the mean such as one standard deviation, two standard deviations and so on. The exact shape of the curve may vary for different metrics such that some curves may be wider or narrower, depending on the variance of system performance for a given metric. Also, the data may not conform entirely to a normal distribution and may be positively or negatively skewed. However, the disclosed system still provides a robust monitoring system when such “imperfect data” exists. For example, second curve 314 and third curve 316 illustrate bell curves having different profiles than first curve 302.
Many system metrics could be utilized to monitor performance. In one embodiment, system performance metrics can detect specific verbal responses such as “operator please” or “I need help” and provide a count of the occurrence of “distress responses” over a given time period. In another embodiment non-responses, lack of confirmations and confidence values associated with caller input can be monitored. Confidence levels can be calculated based on a caller's response to an IVRS prompt. The confidence value for many callers can be accumulated and when such value falls below a threshold level during a given time period the system can also set an alarm. Other performance metrics that could be monitored and processed are “too much speech” or speech input exceeding a certain amount of time, and a high number of unrecognizable phonemes. The source or component creating the alarm can also be logged such that a user may have a better understanding of what portion of the system needs maintenance or repair.
Many mathematical formulas could be utilized to create statistics by weighting different call performance metrics and adding them to others without departing from the scope of the present disclosure. With the disclosed system and method, the call center can avoid prolonged periods of time where the system is underperforming.
Thus, the disclosed method identifies an underperforming or non-performing system quicker, provides an enhanced customer experience, and reduces the quantity of callers having a negative interaction with an automated call support system. An increased level of customer satisfaction can thereby be realized.
The disclosed system and method allows tracking of related human behaviors and may take appropriate actions in various circumstances. The disclosed system and method is applicable to multiple user interface modalities, and can process caller or user input from interactive systems such as websites, touchtone phones, speech recognition systems, and other user interface systems. In addition, the disclosed system and method may lead to a reduced level of abandoned calls (i.e., caller hangs up prior to completing their task), thereby reducing provider system costs (e.g., lower port costs and fewer repeat calls). Some inconsistencies in processing caller inputs are more frustrating to callers than others. The disclosed system and method may provide improved caller satisfaction, since the user on the average can avoid being caught in error-prone situations and frustrating endless loops caused by a broken or impaired automated call processing system.
The above-disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the present invention. Thus, to the maximum extent allowed by law, the scope of the present invention is to be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

1. A method of monitoring an automated call processing system comprising: establishing a performance metric that is related to at least a portion of an

operational characteristic of a call processing center; tracking a performance indicator in near real time, the performance indicator

related to the operational characteristic; detecting a deviation between the performance indicator and the performance

metric; and triggering an alarm in response to the detected deviation.

2. The method of claim 1, wherein the operational characteristic is one of an average call duration, a system response time, a premature disconnect count, a verbal extensible mark up language (VXML) server access time, a caller hang up rate, an excessive confirmation rate, and a false acceptance rate.

3. The method of claim 1, further comprising transmitting a message responsive to the alarm trigger.

4. The method of claim 1, wherein establishing the performance metric further comprises consideration of at least one of a mean a standard deviation, and a z-score developed from a collection of information representing measurements of the operational characteristic.

5. The method of claim 4, wherein the deviation is detected when the performance indicator is greater than or less than plus or minus one (1) standard deviation or a z-score from the mean.

6. The method of claim 1, further comprising comparing the performance indicator and the performance metric on a periodic basis.

7. The method of claim 1, wherein the performance indicator is based on activities related to one of an interactive voice response system, a speech recognition module and a voice extensible mark up language server.

8. The method of claim 1, wherein the performance metric is based on one of an uncorrected mean or a standard deviation.

9. The method of claim 1, wherein the alarm is triggered after one of an extreme value or a time averaged parameters.

10. The method of claim 1, wherein a subsequent caller is routed to an agent when the deviation is detected.

11. A system for monitoring performance of a call center comprising:

an interactive voice response system (IVRS) configured to accept spoken caller input and to process a call responsive to the spoken caller input;

a server coupled to the IVRS and configured to supply executable objects to the IVRS; and

a monitor coupled to the IVRS and configured to monitor at least one parameter of the IVRS related to caller activity, to maintain a performance metric for the at least one parameter, and to determine if the at least one parameter has deviated from the performance metric.

12. The system as in claim 11, wherein the parameter is one of a call time duration, a caller hang up, and a delay time.

13. The system as in claim 11, further comprising a database coupled to the monitor and configured to store information associated with the monitored parameter.

14. The system as in claim 13, further comprising a statistical processor configured to calculate one of an actual mean, an actual standard deviation, an actual range, or an actual median, for the stored information.

15. The system as in claim 14, further comprising calculating one of the actual mean, the actual standard deviation, the actual range, or the actual median, utilizing historical data from approximately a previous half hour time period.

16. The system as in claim 14, further comprising an alarm generator responsive to an output of the statistical processor.

17. The system as in claim 14, further comprising a switch for switching the caller to a human operator when monitored data is not in compliance with the performance metric.

18. A computer readable medium tangibly embodying a program of instructions to manipulate a computing platform to:

establish a baseline performance metric that represents an operational characteristic of a call processing center;

track an actual performance for the operational characteristic in near real time;

detect a deviation between the performance metric and the actual performance; and

trigger an alarm in response to detecting the deviation.

19. The computer readable medium of claim 18, further comprising instructions to process an average operational characteristic including one of an average call duration, a system response time, a premature disconnect count, a voice extensible mark up language (VXML) server access time, a caller hang up rate, an excessive confirmation rate, and a false acceptance rate.

20. The computer readable medium of claim 18, further comprising instructions to transmit a message responsive to the alarm trigger.

21. The computer readable medium of claim 20, further comprising instructions to establish a performance metric by calculating at least one of a mean and a standard deviation, developed from a collection of information representing measurements of the operational characteristic.