Betriebswirtschaftliche Publikationen - Gottfried Rudorfer's Homepage
Offizielle Gottfried Rudorfer's Homepage
https://rudorfer.homedns.org/at/betriebswirtschaftslehre/betriebswirtschaftliche-publikationen
2024-03-29T11:24:22+00:00
Gottfried Rudorfer's Homepage
gottfried@rudorfer.homedns.org
Joomla! - Open Source Content Management
Master Thesis of Gottfried Rudorfer
2010-09-13T19:26:51+00:00
2010-09-13T19:26:51+00:00
https://rudorfer.homedns.org/at/betriebswirtschaftslehre/betriebswirtschaftliche-publikationen/185-master-thesis-of-gottfried-rudorfer
Gottfried Rudorfer
gottfried@rudorfer.homedns.org
<p style="text-align: center;"><span style="font-size: 12pt;"><strong>Master Thesis of Gottfried Rudorfer</strong></span></p>
<p>Title: Anwendung künstlicher Neuronaler Netzwerke als datengetriebenes Analysewerkzeug.</p>
<p>The master thesis is physically available at the library of the University of Economics and Business administration <a target="_blank" href="http://permalink.obvsg.at/wuw/AC00743182">http://permalink.obvsg.at/wuw/AC00743182</a> .</p>
<p>The <a target="_blank" href="http://rudorfer.homedns.org/dipl/dipl.pdf">master thesis</a> is available here.</p>
<p style="text-align: center;"><span style="font-size: 12pt;"><strong>Master Thesis of Gottfried Rudorfer</strong></span></p>
<p>Title: Anwendung künstlicher Neuronaler Netzwerke als datengetriebenes Analysewerkzeug.</p>
<p>The master thesis is physically available at the library of the University of Economics and Business administration <a target="_blank" href="http://permalink.obvsg.at/wuw/AC00743182">http://permalink.obvsg.at/wuw/AC00743182</a> .</p>
<p>The <a target="_blank" href="http://rudorfer.homedns.org/dipl/dipl.pdf">master thesis</a> is available here.</p>
Early Bankruptcy Detection Using Neural Networks
2010-09-13T18:42:04+00:00
2010-09-13T18:42:04+00:00
https://rudorfer.homedns.org/at/betriebswirtschaftslehre/betriebswirtschaftliche-publikationen/186-early-bankruptcy-detection-using-neural-networks
Gottfried Rudorfer
gottfried@rudorfer.homedns.org
<p>Gottfried Rudorfer, Early Bankruptcy Detection Using Neural Networks,<br /> APL Quote Quad, ACM New York, Volume 25, Number 4, June 1995, pages 171-178</p>
<p align="CENTER">Early Bankruptcy Detection Using Neural Networks</p>
<p align="CENTER"><strong>Gottfried Rudorfer</strong></p>
<p align="CENTER"><strong>Department of Applied Computer Science</strong></p>
<p align="CENTER"><strong>University of Economics and Business Administration</strong></p>
<p align="CENTER"><strong>Augasse 2-6</strong></p>
<p align="CENTER"><strong>A-1090 Vienna</strong></p>
<p align="CENTER"><strong>Austria</strong></p>
<p align="CENTER"><a href="mailto:Rudorfer@a1.net"><strong><strong> </strong></strong></a><strong><strong><a href="mailto:gottfried@rudorfer.homedns.org?subject=Early%20Bankruptcy%20Detection%20Using%20Neural%20Networks">gottfried@rudorfer.homedns.org</a></strong></strong></p>
<p> </p>
<dir></dir><hr />
<p><strong>Download the <a href="http://rudorfer.homedns.org/apl95/apl95.pdf">PDF-version</a> here. </strong></p>
<hr />
<p><strong>Abstract</strong></p>
<p> </p>
<p align="JUSTIFY">In 1993, Austria had the highest number of bankruptcies since 1945. The total liabilities came to approx. US$ 3 Billion.</p>
<p align="JUSTIFY">Powerful tools for early detection of company risks are very important to avoid high economic losses. Artificial neural networks(ANN) are suitable for many tasks in pattern recognition and machine learning. In this paper we present an ANN for early detection of company failures using balance sheet ratios. The network has been successfully tested with real data of Austrian private limited companies. The research activities included the design of an APL application with a graphical user interface to find out the relevant input data and tune the ANN.</p>
<p align="JUSTIFY">The developed APL workspace takes advantage of modern windowing features running on IBM compatible computers.</p>
<p align="JUSTIFY"><strong>Keywords</strong> Artificial Neural Networks, Backpropagation, Discriminant Analysis, Bankruptcy, Balance Sheet Ratios, APL.</p>
<dir>
<p><strong>Motivation</strong></p>
</dir>
<p align="JUSTIFY">Neural nets are currently being used to solve problems in many different research areas. Generally, they tend to assist or become an alternate way for traditional statistical and mathematical models.</p>
<p align="JUSTIFY">Their main practical applications in economics have been for time series forecasting (i.e. [KR94]) and classification tasks (i.e. [TK90]).</p>
<p align="JUSTIFY">Banks usually check the creditworthiness of companies to find the maximum amount of credit they are prepared to grant. The models used are formulated as a classification problem in a multidimensional space defined by a set of financial ratios calculated using the balance sheet.</p>
<p align="JUSTIFY">In this paper we try to use neural networks as replacement for the widely used statistical discriminant analysis to early detect company failures. With this approach we bypass the disadvantages and problems with statistical discriminant analysis:</p>
<p align="JUSTIFY">The major problem with statistical discriminant analysis is the assumption of a multivariate normal distribution of the sample data. This assumption is very often violated in practical problems.</p>
<p align="JUSTIFY">We use multilayer feedforward networks also known as multilayered perceptrons (MLPs), because they are a class of universal approximators.</p>
<p align="JUSTIFY">There exists a formal proof that "standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired accuracy, provided sufficiently many hidden units are available". [HSW89].</p>
<p align="JUSTIFY">The networks under investigation have the following properties:</p>
<ul>
<li>the net consists of many simple processing devices grouped into layers</li>
<li>one input/output pair is randomly selected and passed to the artificial neural network after transforming the data</li>
<li>the activation levels are propagated starting at the input layer through the hidden layers to the output layer</li>
<li>the activation level of the neurons in the output layer is fed to the environment</li>
<li>at least on hidden layer is necessary to be able to use the net as a universal approximator</li>
</ul>
<p align="JUSTIFY">Discriminant analysis tries to find a linear subspace of the given patterns such that projections of the patterns onto the subspace are grouped into well separated clusters. In our case, balance sheet ratios are used to assign companies into two clusters, one of sound and one for insolvent companies.</p>
<p align="JUSTIFY">The relationships between statistical discriminant analysis and multilayered perceptrons (MLPs) shows the evidence of generic properties of MLPs classifiers [GTBFS91]. In other words, linear MLPs can be trained to perform mean square classification to discriminant analysis. The authors of the cited paper [GTBFS91] first concentrate on linear MLPs with one hidden layer. Such networks have two weight matrices W1 for the weights from the input-to-hidden and W2 for the weights from the hidden-to-output layers. These two weight matrices are adapted by the backpropagation learning algorithm. If this algorithm minimizes the error between desired and network output vectors, the first layer of the MLP performs discriminant analysis projection using the weights from input-to-hidden W1 and the second layer performs a classification on the output of the hidden units using the weights from hidden-to-output layers.</p>
<p align="JUSTIFY">In this application neural nets perform statistical analysis of data. Therefore they require a large set of data to yield optimal estimation of the parameters. This also implies that the number of given patterns is larger than the dimension of the space in which they sit. Each cluster should be described by enough patterns belonging to that cluster.</p>
<dir>
<p><strong>Implementation</strong></p>
</dir>
<p align="JUSTIFY">The heart of your forecasting system is a "multilayer feedforward network", also known as "multilayered perceptron" or "feed forward networks" trained with the "generalized delta rule", also known as "backpropagation" [RHW86]. For a detailed description of this type of network see [HKP91] and [Nil90] and their APL implementation see [KR94], [Alf91], [ES91], [Pee81] and [SS93]. It was fist implemented using Dyalog APL version 6.1 release 3 on HP9000/700 workstations [Dya91]. The code implementing the graphical user interface (GUI) was then ported to Dyalog APL Version 7.0 release 1 for MS-Windows on IBM compatible PCs [Dya94].</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image13.gif" alt="" width="500" height="400" /></p>
<p><strong>Figure 1: User interface of the simulator</strong></p>
<p align="JUSTIFY"> </p>
<p align="JUSTIFY">The system consists of the following components:</p>
<ul>
<li>The runtime version of the APL interpreter. With this APL dialect the vendor allows distribution of the runtime interpreter version free of charge.</li>
<li>The APL ANN workspace implements the classification system. The functions in the workspace are grouped into the following namespaces:
<ul>
<li>Functions that create graphical objects like a window showing the multilayered perceptron network (see Figure 1). Often other than default properties are specified in GUI functions. Properties define an object's behavior, appearance and events that the object can generate. For each object able to generate events, a "callback function" can be assigned that is called when the corresponding event occurs. Events are very important for an application with a graphical front-end because they define how the objects react to the user. As an example, we show the APL code that allows the user to move the processing devices within the window.</li>
<li>Functions for component file input and output. Because the execution time of such simulations is very high, logging facilities of the application must be provided. The application takes extensively advantage of nested vectors and matrices. With the component filesystem, APL variables can be created/appended directly to the hard disk.</li>
<li>The last but not least part is the APL artificial neural network-code as described in [KR94]. The code consists of three individual parts: During network training, a main loop is used to present many input patterns to the artificial neural network. Within the loop there is a function that reads in and transforms the training data. The "FORWARD" function propagates the input pattern through the hidden layer(s) to the output layer. After the output value of the net is known, an error is calculated and the weight change is propagated backwards through the hidden layers to the input layer using the "BACKWARD" function.</li>
</ul>
</li>
</ul>
<p align="JUSTIFY">The system is generally controlled via the graphical frontend because there is no session manager in the runtime version of the interpreter available.</p>
<p align="JUSTIFY">Figure 1 shows the window of the "Multiple-Document Interface (MDI)" application. The MDI window consists of a menu bar and a tool bar at the top and a status field at the bottom. Within the main window, there are windows for general control of the system and for displaying details of the artificial neural network. The backprop view shows the activation level of the neurons (circles) and the weight values (lines). The window dump shows a net with 5 input, 3 hidden units and 1 output unit (5-3-1). A big filled circle indicates that the neuron fires and a transparent circle indicates (almost) no output of the neuron. The size of the circle is proportional to the output of a neuron. The lines connecting to each neuron from the previous layer are the graphical representation of the weight values. These can be negative or positive real numbers, which are encoded in two different colors. Again, a thick line represents a weight with a high negative or positive value.</p>
<dir>
<p><strong>Modeling</strong></p>
</dir>
<p><strong>The Training Data</strong></p>
<p align="JUSTIFY">In order to indicate more practical relevance, we decided to use real data. First, we studied the insolvency statistics of the year 1993 [Hie93]:</p>
<ul>
<li>total amount of insolvencies: 5,082 (+38 %)</li>
<li>total amount of obligations: US$ 2.9 Billion</li>
<li>17,000 employees evolved</li>
</ul>
<p align="JUSTIFY">In 1994, the "Atomic" company, one of the world's biggest producer of skiis became insolvent. Until now, the year 1994 seems to be another bad year of many insolvencies.</p>
<p align="JUSTIFY">If we split up the total number of insolvencies according to types of company, we get the following statistics [Hie93]:</p>
<table border="1" style="width: 284px;" cellspacing="1" cellpadding="7">
<tbody>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">type of company</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">percentage of total</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">sole trader/partnership</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">33.87</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">unlimited companies</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">1.32</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">private limited companies</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">50.66</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">public limited companies</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">0.44</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">other</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">13.71</span></p>
</td>
</tr>
</tbody>
</table>
<p> </p>
<p><strong>Table 1: Which company types have a high risk to become insolvent?</strong></p>
<p align="JUSTIFY">Another important information is the membership of an individual company in the branch of industry. If we consider the branch of industry, the classification capability will probably be better. The metal-processing, paper, building and construction and the textile industry are the most jeopardized area of business in Austria.</p>
<p align="JUSTIFY">After studying the possible sources of data about insolvent and sound companies, we decided to only survey private limited companies with a minimum turnover of US$ 30,000. Firstly private limited companies tend to become more often insolvent than other company types. Secondly private limited companies with more than ATS 1 Million common share or more than 300 employees have to publish their balance sheet.</p>
<p align="JUSTIFY">What relationships between balance sheet items should be calculated to best describe the financial position of a business firm?</p>
<p align="JUSTIFY">We decided to calculate the following five financial ratios to train our artificial neural network[Ble85]:</p>
<ol>
<li>cash flow / liabilities</li>
<li value="1">quick (current) assets / current liabilities</li>
<li value="1">quick (current) assets / total assets</li>
<li value="1">liabilities / total assets</li>
<li value="1">profit or loss / total assets</li>
</ol>
<p align="JUSTIFY">As test bed for our simulations we used 82 balance sheets, 59 of sound and 23 of insolvent companies. In order to average statistical outliers, we tried to get balance sheets of three successive years from each business firm. The 82 balance sheets were grouped in 62 training- and 20 test patterns.</p>
<p align="JUSTIFY">The following figures show the distribution of the parameters:</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image25.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 2: Distribution of balance sheet ratio cash flow / liabilities</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image26.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 3: Distribution of balance sheet ratio quick (current) assets / current liabilities</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image16.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 4: Distribution of balance sheet ratio quick (current) assets / total assets</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image27.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 5: Distribution of balance sheet ratio liabilities / total assets</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image28.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 6: Distribution of balance sheet ratio profit or loss / total assets</strong></p>
<p> </p>
<p><strong>The Algorithm</strong></p>
<p align="JUSTIFY">During the learning phase a randomly selected input/output pair is presented to the network. The input/output data is transformed into a range that the ANN can process. The output values are set to 1 for insolvent companies and to 0 for all others. The simulation environment helps the researcher to find a well performing network. Typically a session is divided into the following steps:</p>
<ul>
<li>Select a file for saving the network status in the file box menu</li>
<li>Load the training data with the input/output pattern selection menu</li>
<li>Set the network parameters (learning rate, momentum term, output function, maximum number of iterations) in the control menu</li>
<li>Start learning</li>
<li>Load test patterns and test the network</li>
</ul>
<p align="JUSTIFY">We have conducted many experiments to find the size of a well performing network. Table 2 and Figure 7 show the results of different network topologies in the test data sample after 5,000 learning iterations, using a learning rate of 0.3 and a momentum of 0.8.</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image29.gif" alt="" width="50%" height="50%" /></p>
<p align="JUSTIFY"><strong>Table 2: Results for various network topologies</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image30.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 7: The performance of different network topologies</strong></p>
<p align="JUSTIFY">Because the network output has values between 0 and 1, we have to introduce a threshold to assign a company to the insolvent or sound cluster. For a threshold of 0.5, the 5-3-1 and 5-5-1 networks have the best performance with one wrong classification of an insolvent company (number 18).</p>
<p align="JUSTIFY">We have chosen the smaller network for closer inspection.</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image31.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 8: Graphical view of the found 5-3-1 network</strong></p>
<p> </p>
<p align="JUSTIFY">The weight matrix between input (L0) and hidden (L1) layer is given in Table 3:</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image32.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Table 3: The weight matrix between input and hidden layer</strong></p>
<p align="JUSTIFY">The weight vector between hidden (L1) and output (L2) layer is shown in Table 4:</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image33.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Table 4: Weight vector between hidden and output layer</strong></p>
<p align="JUSTIFY">The Data was transformed with linear functions (see Table 5).</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image34.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Table 5: Minimum and maximum values of original and transformed data</strong></p>
<p align="JUSTIFY">One neuron in the hidden layer is a detector for a sound and another for an insolvent company. The third neuron has a very small impact on the final result because of small weight values.</p>
<p align="JUSTIFY">A company seems to be in danger when the ratio liabilities / total assets or quick assets / assets has a high positive value. On the other side, sound companies have small liabilities / total assets and quick assets / assets ratios.</p>
<p><strong>Dyalog APL GUI Code</strong></p>
<p align="JUSTIFY">In this paper we want to present a piece of code, that implements the graphical user interface (GUI). The following functions implement the neuron (circles in figure) movable functionality. This enables the user to reposition the processing devices in the "backprop view" window. The circle is redrawn by APL itself. Only the weights (lines) connected to the neuron needs to repositioned. This is done with the function MOVE.</p>
<p align="JUSTIFY"> </p>
<p><img src="http://rudorfer.homedns.org/apl95/Image35.gif" alt="" width="538" height="246" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image36.gif" alt="" width="824" height="381" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image37.gif" alt="" width="736" height="91" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image38.gif" alt="" width="881" height="169" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image39.gif" alt="" width="655" height="153" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image40.gif" alt="" width="794" height="227" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image41.gif" alt="" width="599" height="238" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image42.gif" alt="" width="732" height="216" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image43.gif" alt="" width="544" height="102" /></p>
<dir>
<p><strong>Conclusion</strong></p>
</dir>
<p align="JUSTIFY">In this paper we have presented an APL tool for early detection of company failures using neural networks. This computing devices proved themselves to be a viable alternative to discriminant analysis.</p>
<p align="JUSTIFY">With this workspace we were able to find networks for detection of company failures. Graphical visualization helps understanding complex relations that exist in neural nets.</p>
<p align="JUSTIFY">Further work will include the improvement of the APL product and a detailed documentation of the user interface.</p>
<dir>
<p><strong>References</strong></p>
</dir>
<p align="JUSTIFY">[Alf91] M. Alfonseca. Advanced applications of APL: logic programming, neural networks and hypertext. IBM Systems Journal, 30(4):543-553, 1991.</p>
<p align="JUSTIFY">[Ble85] Ernst Bleier. Insolvenzfrüherkennung mittels praktischer Anwendung der Diskriminanzanalyse. Service Fachverlag an der Wirtschaftsuniversität Wien, Augasse 2-6, 1090 Vienna, Austria, 1985.</p>
<p align="JUSTIFY">[Dya91] Dyadic Systems Limited, Riverside View, Basing Road, Old Basing, Basingstoke, Hampshire RG24 0AL, England. Dyalog Apl Users Guide for version 6.1, 1991.</p>
<p align="JUSTIFY">[Dya94] Dyadic Systems Limited, Riverside View, Basing Road, Old Basing, Basingstoke, Hampshire RG24 0AL, England. Dyalog Apl User Guide, Language Reference, Windows Interface and Outer Products for Version 7.0, 1994.</p>
<p align="JUSTIFY">[ES91] Richard M. Evans and Alvin J. Surkan. Relating Numbers of Processing Elements in a Sparse Distributed Memory Model to Learning Rate and Generalization. ACM APL Quote Quad, 21(4):166-173, 1991.</p>
<p align="JUSTIFY">[GTBFS91] P. Gallinari, S. Thiria, F. Badran, and F. Folgelman-Soulie. On the Relations Between Discriminant Analysis and Multilayer Perceptrons. Neural Networks, 4:349-360, 1991.</p>
<p align="JUSTIFY">[Hie93] Klaus Hierzenberger. Bericht zur Insovenzstatistik 1993. Kreditschutzverband von 1870, 1993.</p>
<p align="JUSTIFY">[HKP91] John Hertz, Anders Krogh, and Richard G. Palmer. Introduction to the Theory od Neural Computation. Addison Wesley, Redwood City, California, 1991.</p>
<p align="JUSTIFY">[HSW89] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer Feedforward Networks are Universal Approximators. Neural Networks, 2:359-366, 1989.</p>
<p align="JUSTIFY">[KR94] Thomas Kolarik and Gottfried Rudorfer. Time Series Forecasting Using Neural Networks. ACM APL Quote Quad, 25(1):86-94, 1994.</p>
<p align="JUSTIFY">[Nil90] Nils J. Nilsson. The Mathematical Foundations of Learning Machines. Morgan Kaufmann Publishers Inc., San Mateo, 1990.</p>
<p align="JUSTIFY">[Pee81] Howard A. Peele. Teaching A Topic in Cybernetics with APL: An Introduction to Neural Net Modelling. ACM APL Quote Quad, 12(1):235-239, 1981.</p>
<p align="JUSTIFY">[RHW86] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by backpropagating errors. Nature, 323(9):533-536, October 1986.</p>
<p align="JUSTIFY">[RW94] Gottfried Rudorfer and Harald Wenisch. Isolvenzprognose mit Künstlichen Neuronalen Netzen. University of Economics and Business Administration, Augasse 2-6, 1090 Wienna, Austria, 1994.</p>
<p align="JUSTIFY">[SS93] Alexei N. Skurihin and Alvin J. Surkan. Identification of Parallelism in Neural Networks by Simulation with Language J. ACM APL Quote Quad, 24(1):230-237, 1993.</p>
<p align="JUSTIFY">[TK90] Kar Yan Tam and Melody Kiang. Predicting Bank Failures: A Neural Network Approach. Applied Artificial Intelligence, 4:265-282, 1990.</p>
<hr />
<p>Gottfried Rudorfer, Early Bankruptcy Detection Using Neural Networks,<br /> APL Quote Quad, ACM New York, Volume 25, Number 4, June 1995, pages 171-178</p>
<p align="CENTER">Early Bankruptcy Detection Using Neural Networks</p>
<p align="CENTER"><strong>Gottfried Rudorfer</strong></p>
<p align="CENTER"><strong>Department of Applied Computer Science</strong></p>
<p align="CENTER"><strong>University of Economics and Business Administration</strong></p>
<p align="CENTER"><strong>Augasse 2-6</strong></p>
<p align="CENTER"><strong>A-1090 Vienna</strong></p>
<p align="CENTER"><strong>Austria</strong></p>
<p align="CENTER"><a href="mailto:Rudorfer@a1.net"><strong><strong> </strong></strong></a><strong><strong><a href="mailto:gottfried@rudorfer.homedns.org?subject=Early%20Bankruptcy%20Detection%20Using%20Neural%20Networks">gottfried@rudorfer.homedns.org</a></strong></strong></p>
<p> </p>
<dir></dir><hr />
<p><strong>Download the <a href="http://rudorfer.homedns.org/apl95/apl95.pdf">PDF-version</a> here. </strong></p>
<hr />
<p><strong>Abstract</strong></p>
<p> </p>
<p align="JUSTIFY">In 1993, Austria had the highest number of bankruptcies since 1945. The total liabilities came to approx. US$ 3 Billion.</p>
<p align="JUSTIFY">Powerful tools for early detection of company risks are very important to avoid high economic losses. Artificial neural networks(ANN) are suitable for many tasks in pattern recognition and machine learning. In this paper we present an ANN for early detection of company failures using balance sheet ratios. The network has been successfully tested with real data of Austrian private limited companies. The research activities included the design of an APL application with a graphical user interface to find out the relevant input data and tune the ANN.</p>
<p align="JUSTIFY">The developed APL workspace takes advantage of modern windowing features running on IBM compatible computers.</p>
<p align="JUSTIFY"><strong>Keywords</strong> Artificial Neural Networks, Backpropagation, Discriminant Analysis, Bankruptcy, Balance Sheet Ratios, APL.</p>
<dir>
<p><strong>Motivation</strong></p>
</dir>
<p align="JUSTIFY">Neural nets are currently being used to solve problems in many different research areas. Generally, they tend to assist or become an alternate way for traditional statistical and mathematical models.</p>
<p align="JUSTIFY">Their main practical applications in economics have been for time series forecasting (i.e. [KR94]) and classification tasks (i.e. [TK90]).</p>
<p align="JUSTIFY">Banks usually check the creditworthiness of companies to find the maximum amount of credit they are prepared to grant. The models used are formulated as a classification problem in a multidimensional space defined by a set of financial ratios calculated using the balance sheet.</p>
<p align="JUSTIFY">In this paper we try to use neural networks as replacement for the widely used statistical discriminant analysis to early detect company failures. With this approach we bypass the disadvantages and problems with statistical discriminant analysis:</p>
<p align="JUSTIFY">The major problem with statistical discriminant analysis is the assumption of a multivariate normal distribution of the sample data. This assumption is very often violated in practical problems.</p>
<p align="JUSTIFY">We use multilayer feedforward networks also known as multilayered perceptrons (MLPs), because they are a class of universal approximators.</p>
<p align="JUSTIFY">There exists a formal proof that "standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired accuracy, provided sufficiently many hidden units are available". [HSW89].</p>
<p align="JUSTIFY">The networks under investigation have the following properties:</p>
<ul>
<li>the net consists of many simple processing devices grouped into layers</li>
<li>one input/output pair is randomly selected and passed to the artificial neural network after transforming the data</li>
<li>the activation levels are propagated starting at the input layer through the hidden layers to the output layer</li>
<li>the activation level of the neurons in the output layer is fed to the environment</li>
<li>at least on hidden layer is necessary to be able to use the net as a universal approximator</li>
</ul>
<p align="JUSTIFY">Discriminant analysis tries to find a linear subspace of the given patterns such that projections of the patterns onto the subspace are grouped into well separated clusters. In our case, balance sheet ratios are used to assign companies into two clusters, one of sound and one for insolvent companies.</p>
<p align="JUSTIFY">The relationships between statistical discriminant analysis and multilayered perceptrons (MLPs) shows the evidence of generic properties of MLPs classifiers [GTBFS91]. In other words, linear MLPs can be trained to perform mean square classification to discriminant analysis. The authors of the cited paper [GTBFS91] first concentrate on linear MLPs with one hidden layer. Such networks have two weight matrices W1 for the weights from the input-to-hidden and W2 for the weights from the hidden-to-output layers. These two weight matrices are adapted by the backpropagation learning algorithm. If this algorithm minimizes the error between desired and network output vectors, the first layer of the MLP performs discriminant analysis projection using the weights from input-to-hidden W1 and the second layer performs a classification on the output of the hidden units using the weights from hidden-to-output layers.</p>
<p align="JUSTIFY">In this application neural nets perform statistical analysis of data. Therefore they require a large set of data to yield optimal estimation of the parameters. This also implies that the number of given patterns is larger than the dimension of the space in which they sit. Each cluster should be described by enough patterns belonging to that cluster.</p>
<dir>
<p><strong>Implementation</strong></p>
</dir>
<p align="JUSTIFY">The heart of your forecasting system is a "multilayer feedforward network", also known as "multilayered perceptron" or "feed forward networks" trained with the "generalized delta rule", also known as "backpropagation" [RHW86]. For a detailed description of this type of network see [HKP91] and [Nil90] and their APL implementation see [KR94], [Alf91], [ES91], [Pee81] and [SS93]. It was fist implemented using Dyalog APL version 6.1 release 3 on HP9000/700 workstations [Dya91]. The code implementing the graphical user interface (GUI) was then ported to Dyalog APL Version 7.0 release 1 for MS-Windows on IBM compatible PCs [Dya94].</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image13.gif" alt="" width="500" height="400" /></p>
<p><strong>Figure 1: User interface of the simulator</strong></p>
<p align="JUSTIFY"> </p>
<p align="JUSTIFY">The system consists of the following components:</p>
<ul>
<li>The runtime version of the APL interpreter. With this APL dialect the vendor allows distribution of the runtime interpreter version free of charge.</li>
<li>The APL ANN workspace implements the classification system. The functions in the workspace are grouped into the following namespaces:
<ul>
<li>Functions that create graphical objects like a window showing the multilayered perceptron network (see Figure 1). Often other than default properties are specified in GUI functions. Properties define an object's behavior, appearance and events that the object can generate. For each object able to generate events, a "callback function" can be assigned that is called when the corresponding event occurs. Events are very important for an application with a graphical front-end because they define how the objects react to the user. As an example, we show the APL code that allows the user to move the processing devices within the window.</li>
<li>Functions for component file input and output. Because the execution time of such simulations is very high, logging facilities of the application must be provided. The application takes extensively advantage of nested vectors and matrices. With the component filesystem, APL variables can be created/appended directly to the hard disk.</li>
<li>The last but not least part is the APL artificial neural network-code as described in [KR94]. The code consists of three individual parts: During network training, a main loop is used to present many input patterns to the artificial neural network. Within the loop there is a function that reads in and transforms the training data. The "FORWARD" function propagates the input pattern through the hidden layer(s) to the output layer. After the output value of the net is known, an error is calculated and the weight change is propagated backwards through the hidden layers to the input layer using the "BACKWARD" function.</li>
</ul>
</li>
</ul>
<p align="JUSTIFY">The system is generally controlled via the graphical frontend because there is no session manager in the runtime version of the interpreter available.</p>
<p align="JUSTIFY">Figure 1 shows the window of the "Multiple-Document Interface (MDI)" application. The MDI window consists of a menu bar and a tool bar at the top and a status field at the bottom. Within the main window, there are windows for general control of the system and for displaying details of the artificial neural network. The backprop view shows the activation level of the neurons (circles) and the weight values (lines). The window dump shows a net with 5 input, 3 hidden units and 1 output unit (5-3-1). A big filled circle indicates that the neuron fires and a transparent circle indicates (almost) no output of the neuron. The size of the circle is proportional to the output of a neuron. The lines connecting to each neuron from the previous layer are the graphical representation of the weight values. These can be negative or positive real numbers, which are encoded in two different colors. Again, a thick line represents a weight with a high negative or positive value.</p>
<dir>
<p><strong>Modeling</strong></p>
</dir>
<p><strong>The Training Data</strong></p>
<p align="JUSTIFY">In order to indicate more practical relevance, we decided to use real data. First, we studied the insolvency statistics of the year 1993 [Hie93]:</p>
<ul>
<li>total amount of insolvencies: 5,082 (+38 %)</li>
<li>total amount of obligations: US$ 2.9 Billion</li>
<li>17,000 employees evolved</li>
</ul>
<p align="JUSTIFY">In 1994, the "Atomic" company, one of the world's biggest producer of skiis became insolvent. Until now, the year 1994 seems to be another bad year of many insolvencies.</p>
<p align="JUSTIFY">If we split up the total number of insolvencies according to types of company, we get the following statistics [Hie93]:</p>
<table border="1" style="width: 284px;" cellspacing="1" cellpadding="7">
<tbody>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">type of company</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">percentage of total</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">sole trader/partnership</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">33.87</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">unlimited companies</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">1.32</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">private limited companies</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">50.66</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">public limited companies</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">0.44</span></p>
</td>
</tr>
<tr>
<td valign="TOP" width="58%">
<p><span style="font-size: 8pt;">other</span></p>
</td>
<td valign="TOP" width="42%">
<p><span style="font-size: 8pt;">13.71</span></p>
</td>
</tr>
</tbody>
</table>
<p> </p>
<p><strong>Table 1: Which company types have a high risk to become insolvent?</strong></p>
<p align="JUSTIFY">Another important information is the membership of an individual company in the branch of industry. If we consider the branch of industry, the classification capability will probably be better. The metal-processing, paper, building and construction and the textile industry are the most jeopardized area of business in Austria.</p>
<p align="JUSTIFY">After studying the possible sources of data about insolvent and sound companies, we decided to only survey private limited companies with a minimum turnover of US$ 30,000. Firstly private limited companies tend to become more often insolvent than other company types. Secondly private limited companies with more than ATS 1 Million common share or more than 300 employees have to publish their balance sheet.</p>
<p align="JUSTIFY">What relationships between balance sheet items should be calculated to best describe the financial position of a business firm?</p>
<p align="JUSTIFY">We decided to calculate the following five financial ratios to train our artificial neural network[Ble85]:</p>
<ol>
<li>cash flow / liabilities</li>
<li value="1">quick (current) assets / current liabilities</li>
<li value="1">quick (current) assets / total assets</li>
<li value="1">liabilities / total assets</li>
<li value="1">profit or loss / total assets</li>
</ol>
<p align="JUSTIFY">As test bed for our simulations we used 82 balance sheets, 59 of sound and 23 of insolvent companies. In order to average statistical outliers, we tried to get balance sheets of three successive years from each business firm. The 82 balance sheets were grouped in 62 training- and 20 test patterns.</p>
<p align="JUSTIFY">The following figures show the distribution of the parameters:</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image25.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 2: Distribution of balance sheet ratio cash flow / liabilities</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image26.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 3: Distribution of balance sheet ratio quick (current) assets / current liabilities</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image16.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 4: Distribution of balance sheet ratio quick (current) assets / total assets</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image27.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 5: Distribution of balance sheet ratio liabilities / total assets</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image28.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 6: Distribution of balance sheet ratio profit or loss / total assets</strong></p>
<p> </p>
<p><strong>The Algorithm</strong></p>
<p align="JUSTIFY">During the learning phase a randomly selected input/output pair is presented to the network. The input/output data is transformed into a range that the ANN can process. The output values are set to 1 for insolvent companies and to 0 for all others. The simulation environment helps the researcher to find a well performing network. Typically a session is divided into the following steps:</p>
<ul>
<li>Select a file for saving the network status in the file box menu</li>
<li>Load the training data with the input/output pattern selection menu</li>
<li>Set the network parameters (learning rate, momentum term, output function, maximum number of iterations) in the control menu</li>
<li>Start learning</li>
<li>Load test patterns and test the network</li>
</ul>
<p align="JUSTIFY">We have conducted many experiments to find the size of a well performing network. Table 2 and Figure 7 show the results of different network topologies in the test data sample after 5,000 learning iterations, using a learning rate of 0.3 and a momentum of 0.8.</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image29.gif" alt="" width="50%" height="50%" /></p>
<p align="JUSTIFY"><strong>Table 2: Results for various network topologies</strong></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image30.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 7: The performance of different network topologies</strong></p>
<p align="JUSTIFY">Because the network output has values between 0 and 1, we have to introduce a threshold to assign a company to the insolvent or sound cluster. For a threshold of 0.5, the 5-3-1 and 5-5-1 networks have the best performance with one wrong classification of an insolvent company (number 18).</p>
<p align="JUSTIFY">We have chosen the smaller network for closer inspection.</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image31.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Figure 8: Graphical view of the found 5-3-1 network</strong></p>
<p> </p>
<p align="JUSTIFY">The weight matrix between input (L0) and hidden (L1) layer is given in Table 3:</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image32.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Table 3: The weight matrix between input and hidden layer</strong></p>
<p align="JUSTIFY">The weight vector between hidden (L1) and output (L2) layer is shown in Table 4:</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image33.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Table 4: Weight vector between hidden and output layer</strong></p>
<p align="JUSTIFY">The Data was transformed with linear functions (see Table 5).</p>
<p><img src="http://rudorfer.homedns.org/apl95/Image34.gif" alt="" /></p>
<p align="JUSTIFY"><strong>Table 5: Minimum and maximum values of original and transformed data</strong></p>
<p align="JUSTIFY">One neuron in the hidden layer is a detector for a sound and another for an insolvent company. The third neuron has a very small impact on the final result because of small weight values.</p>
<p align="JUSTIFY">A company seems to be in danger when the ratio liabilities / total assets or quick assets / assets has a high positive value. On the other side, sound companies have small liabilities / total assets and quick assets / assets ratios.</p>
<p><strong>Dyalog APL GUI Code</strong></p>
<p align="JUSTIFY">In this paper we want to present a piece of code, that implements the graphical user interface (GUI). The following functions implement the neuron (circles in figure) movable functionality. This enables the user to reposition the processing devices in the "backprop view" window. The circle is redrawn by APL itself. Only the weights (lines) connected to the neuron needs to repositioned. This is done with the function MOVE.</p>
<p align="JUSTIFY"> </p>
<p><img src="http://rudorfer.homedns.org/apl95/Image35.gif" alt="" width="538" height="246" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image36.gif" alt="" width="824" height="381" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image37.gif" alt="" width="736" height="91" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image38.gif" alt="" width="881" height="169" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image39.gif" alt="" width="655" height="153" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image40.gif" alt="" width="794" height="227" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image41.gif" alt="" width="599" height="238" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image42.gif" alt="" width="732" height="216" /></p>
<p><img src="http://rudorfer.homedns.org/apl95/Image43.gif" alt="" width="544" height="102" /></p>
<dir>
<p><strong>Conclusion</strong></p>
</dir>
<p align="JUSTIFY">In this paper we have presented an APL tool for early detection of company failures using neural networks. This computing devices proved themselves to be a viable alternative to discriminant analysis.</p>
<p align="JUSTIFY">With this workspace we were able to find networks for detection of company failures. Graphical visualization helps understanding complex relations that exist in neural nets.</p>
<p align="JUSTIFY">Further work will include the improvement of the APL product and a detailed documentation of the user interface.</p>
<dir>
<p><strong>References</strong></p>
</dir>
<p align="JUSTIFY">[Alf91] M. Alfonseca. Advanced applications of APL: logic programming, neural networks and hypertext. IBM Systems Journal, 30(4):543-553, 1991.</p>
<p align="JUSTIFY">[Ble85] Ernst Bleier. Insolvenzfrüherkennung mittels praktischer Anwendung der Diskriminanzanalyse. Service Fachverlag an der Wirtschaftsuniversität Wien, Augasse 2-6, 1090 Vienna, Austria, 1985.</p>
<p align="JUSTIFY">[Dya91] Dyadic Systems Limited, Riverside View, Basing Road, Old Basing, Basingstoke, Hampshire RG24 0AL, England. Dyalog Apl Users Guide for version 6.1, 1991.</p>
<p align="JUSTIFY">[Dya94] Dyadic Systems Limited, Riverside View, Basing Road, Old Basing, Basingstoke, Hampshire RG24 0AL, England. Dyalog Apl User Guide, Language Reference, Windows Interface and Outer Products for Version 7.0, 1994.</p>
<p align="JUSTIFY">[ES91] Richard M. Evans and Alvin J. Surkan. Relating Numbers of Processing Elements in a Sparse Distributed Memory Model to Learning Rate and Generalization. ACM APL Quote Quad, 21(4):166-173, 1991.</p>
<p align="JUSTIFY">[GTBFS91] P. Gallinari, S. Thiria, F. Badran, and F. Folgelman-Soulie. On the Relations Between Discriminant Analysis and Multilayer Perceptrons. Neural Networks, 4:349-360, 1991.</p>
<p align="JUSTIFY">[Hie93] Klaus Hierzenberger. Bericht zur Insovenzstatistik 1993. Kreditschutzverband von 1870, 1993.</p>
<p align="JUSTIFY">[HKP91] John Hertz, Anders Krogh, and Richard G. Palmer. Introduction to the Theory od Neural Computation. Addison Wesley, Redwood City, California, 1991.</p>
<p align="JUSTIFY">[HSW89] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer Feedforward Networks are Universal Approximators. Neural Networks, 2:359-366, 1989.</p>
<p align="JUSTIFY">[KR94] Thomas Kolarik and Gottfried Rudorfer. Time Series Forecasting Using Neural Networks. ACM APL Quote Quad, 25(1):86-94, 1994.</p>
<p align="JUSTIFY">[Nil90] Nils J. Nilsson. The Mathematical Foundations of Learning Machines. Morgan Kaufmann Publishers Inc., San Mateo, 1990.</p>
<p align="JUSTIFY">[Pee81] Howard A. Peele. Teaching A Topic in Cybernetics with APL: An Introduction to Neural Net Modelling. ACM APL Quote Quad, 12(1):235-239, 1981.</p>
<p align="JUSTIFY">[RHW86] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by backpropagating errors. Nature, 323(9):533-536, October 1986.</p>
<p align="JUSTIFY">[RW94] Gottfried Rudorfer and Harald Wenisch. Isolvenzprognose mit Künstlichen Neuronalen Netzen. University of Economics and Business Administration, Augasse 2-6, 1090 Wienna, Austria, 1994.</p>
<p align="JUSTIFY">[SS93] Alexei N. Skurihin and Alvin J. Surkan. Identification of Parallelism in Neural Networks by Simulation with Language J. ACM APL Quote Quad, 24(1):230-237, 1993.</p>
<p align="JUSTIFY">[TK90] Kar Yan Tam and Melody Kiang. Predicting Bank Failures: A Neural Network Approach. Applied Artificial Intelligence, 4:265-282, 1990.</p>
<hr />
Time Series Forecasting Using Neural Networks
2010-09-12T21:02:40+00:00
2010-09-12T21:02:40+00:00
https://rudorfer.homedns.org/at/betriebswirtschaftslehre/betriebswirtschaftliche-publikationen/187-time-series-forecasting-using-neural-networks
Gottfried Rudorfer
gottfried@rudorfer.homedns.org
<p style="text-align: center;"><strong><span style="font-size: 14pt;"><em>Time Series Forecasting Using Neural Networks</em></span></strong></p>
<p align="CENTER"><strong><em><big>Thomas Kolarik and Gottfried Rudorfer <br /> <em> <big><span style="font-size: 10pt;">Department of Applied Computer Science </span><br /> <em> <big><span style="font-size: 10pt;">Vienna University of Economics and Business Administration </span><br /> </big></em></big></em></big><em><em>Augasse 2-6, A-1090 Vienna, Austria<big> <big> <big> <big> <br /> </big></big></big></big><tt><a href="mailto:gottfried@rudorfer.homedns.org?subject=Time%20Series%20Forecasting%20Using%20Neural%20Networks">gottfried@rudorfer.homedns.org</a></tt></em></em></em></strong></p>
<hr />
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node1.html#SECTION00010000000000000000" name="tex2html19"> Abstract</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node2.html#SECTION00020000000000000000" name="tex2html20"> Motivation</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node3.html#SECTION00030000000000000000" name="tex2html21"> A Very Short Introduction to Artificial Neural Networks</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node4.html#SECTION00040000000000000000" name="tex2html22"> Implementation</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node5.html#SECTION00050000000000000000" name="tex2html23"> Modeling</a>
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node6.html#SECTION00051000000000000000" name="tex2html24"> The Training Sets</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node7.html#SECTION00052000000000000000" name="tex2html25"> The Algorithm</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node8.html#SECTION00053000000000000000" name="tex2html26"> Network Parameters</a></li>
</ul>
</li>
<li><a href="http://rudorfer.homedns.org/apl94/node9.html#SECTION00060000000000000000" name="tex2html27"> Comparison with ARIMA Modeling</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node10.html#SECTION00070000000000000000" name="tex2html28"> Dyalog APL ANN Code</a>
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node11.html#SECTION00071000000000000000" name="tex2html29"> BACKWARD</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node12.html#SECTION00072000000000000000" name="tex2html30"> FORWARD</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node13.html#SECTION00073000000000000000" name="tex2html31"> GRADIENT</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node14.html#SECTION00074000000000000000" name="tex2html32"> NET</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node15.html#SECTION00075000000000000000" name="tex2html33"> SQUASH</a></li>
</ul>
</li>
<li><a href="http://rudorfer.homedns.org/apl94/node16.html#SECTION00080000000000000000" name="tex2html34"> Conclusion and Further Work</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node17.html#SECTION00090000000000000000" name="tex2html35"> References</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node18.html#SECTION000100000000000000000" name="tex2html36"> About this document ... </a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node18.html#SECTION000100000000000000000" name="tex2html36"></a></li>
</ul>
<h1><a name="SECTION00010000000000000000"> Abstract</a></h1>
<p>Artificial neural networks are suitable for many tasks in pattern recognition and machine learning. In this paper we present an APL system for forecasting univariate time series with artificial neural networks. Unlike conventional techniques for time series analysis, an artificial neural network needs little information about the time series data and can be applied to a broad range of problems. However, the problem of network ``tuning'' remains: parameters of the backpropagation algorithm as well as the network topology need to be adjusted for optimal performance. For our application, we conducted experiments to find the right parameters for a forecasting network. The artificial neural networks that were found delivered a better forecasting performance than results obtained by the well known ARIMA technique.</p>
<h1><a name="SECTION00020000000000000000"> Motivation</a></h1>
<p>Time series analysis as described by most textbooks [<a href="http://rudorfer.homedns.org/apl94/node17.html#chatfield">Cha91</a>] relies on explicit descriptive, stochastic, spectral or other models of processes that describe the real world phenomena generating the observed data.</p>
<p>Usually, the parameters of a standard model like the ARIMA technique [<a href="http://rudorfer.homedns.org/apl94/node17.html#box-jenkins">BJ76</a>] are derived from the autocorrelation and frequency spectrum of the time series. Problems with the ARIMA approach arise with time series of increasing variance or when the time series represents nonlinear processes.</p>
<p>The usage of artificial neural networks for time series analysis relies purely on the data that were observed. As multi layer feed forward networks with at least one hidden layer and a sufficient number of hidden units are capable of approximating any measurable function [<a href="http://rudorfer.homedns.org/apl94/node17.html#hornik">HSW89</a>,<a href="http://rudorfer.homedns.org/apl94/node17.html#turingnn">SS91</a>], an artificial neural network is powerful enough to represent any form of time series. The capability to generalize allows artificial neural networks to learn even in the case of noisy and/or missing data. Another advantage over linear models like the ARIMA technique is the network's ability to represent nonlinear time series.</p>
<p>The APL programming language is very suitable for the task of implementing neural networks [<a href="http://rudorfer.homedns.org/apl94/node17.html#alf91">Alf91</a>,<a href="http://rudorfer.homedns.org/apl94/node17.html#peele81">Pee81</a>,<a href="http://rudorfer.homedns.org/apl94/node17.html#evans91">ES91</a>,<a href="http://rudorfer.homedns.org/apl94/node17.html#sku93">SS93</a>] because of its ability of handle matrix and vector operations. The forward and backward paths of a fully connected feed forward network can be implemented by outer and inner products of vectors and matrices in a few lines of APL code.</p>
<p>For our application, we decided to use a fully connected, layered, feed forward artificial neural network with one hidden layer and the backpropagation learning algorithm. The next section gives a short overview of the relevant definitions and algorithms.</p>
<h1><a name="SECTION00030000000000000000"> A Very Short Introduction to Artificial Neural Networks</a></h1>
<p><a name="chap:intro"> </a> As mentioned above, our simulations utilized the ``multi layered perceptron model'' (MLP), also known as ``feed forward networks'' trained with the ``generalized delta rule'', also known as ``backpropagation''.</p>
<p>The foundations of the backpropagation method for learning in neural networks were laid by [<a href="http://rudorfer.homedns.org/apl94/node17.html#rummel">RHW86</a>].</p>
<p>Artificial neural networks consist of many simple processing devices (called processing elements or neurons) grouped in layers. Each layer is identified by the index <img src="http://rudorfer.homedns.org/apl94/img1.gif" alt="$l=0,\ldots,L$" width="87" height="29" align="MIDDLE" border="0" />. The layers and <em>L</em> are called the ``input layer'' and ``output layer'', all other layers are called ``hidden layers''. The processing elements are interconnected as follows: Communication between processing elements is only allowed for processing elements of neighbouring layers. Neurons within a layer cannot communicate. Each neuron has a certain activation level <em>a</em>. The network processes data by the exchange of activation levels between connected neurons (see figure <a href="http://rudorfer.homedns.org/apl94/node3.html#multi_detail">1</a>):</p>
<p> </p>
<div align="CENTER"><a name="285"> </a>
<table><caption><strong>Figure:</strong> <a name="multi_detail"> </a> Exchange of activation values between neurons</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img2.gif" alt="\begin{figure} \centerline{ \epsfbox {multi_detail.eps} } \end{figure}" width="300" height="610" /></td>
</tr>
</tbody>
</table>
</div>
<p><br /> The output value of the <em>i</em>-th neuron in layer <em>l</em> is denoted by <em>x</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup>. It is calculated with the formula</p>
<p align="CENTER"><em>x</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup> = <em>g</em>(<em>a</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup>)</p>
<p>where <img src="http://rudorfer.homedns.org/apl94/img3.gif" alt="$g(\cdot)$" width="29" height="31" align="MIDDLE" border="0" /> is a monotone increasing function. For our examples, we use the function <img src="http://rudorfer.homedns.org/apl94/img4.gif" alt="$g(y)=\frac{1}{1 + e^{-y}}$" width="99" height="34" align="MIDDLE" border="0" /> (the ``squashing function''). The activation level <em>a</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup> of the neuron <em>i</em> in layer <em>l</em> is calculated by</p>
<p align="CENTER"><em>a</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup> = <em>f</em>(<em>u</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup>)</p>
<p>where <img src="http://rudorfer.homedns.org/apl94/img5.gif" alt="$f(\cdot)$" width="31" height="31" align="MIDDLE" border="0" /> is the activation function (in our case the identity function is used).</p>
<p>The net input <em>u</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup> of neuron <em>i</em> in layer <em>l</em> is calculated as</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img6.gif" alt="\begin{displaymath} u_i^{(l)} = \biggl( \sum_{j = 1}^{n^{(l-1)}} {w_{ij}^{(l)} x_j^{(l-1)} \biggr)} - \Theta_i^{(l)}\end{displaymath}" width="228" height="63" /></p>
<p>where <em>w</em><sub><em>ij</em></sub><sup>(<em>l</em>)</sup> is the weight of neuron <em>j</em> in layer <em>l</em>-1 connected to neuron <em>i</em> in layer <em>l</em>, <em>x</em><sub><em>j</em></sub><sup>(<em>l</em>-1)</sup> is the output of neuron <em>j</em> in layer <em>l</em>-1. <img src="http://rudorfer.homedns.org/apl94/img7.gif" alt="$\Theta_i^{(l)}$" width="32" height="40" align="MIDDLE" border="0" /> is a bias value that is subtracted from the sum of the weighted activations.</p>
<p>The calculation of the network status starts at the input layer and ends at the output layer. The input vector <em>I</em> initializes the activation levels of the neurons in the input layer:</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img8.gif" alt="\begin{displaymath} a^{(0)}_i = {\rm i^{\rm th}~element~of~I}\end{displaymath}" width="159" height="29" /></p>
<p>For the input layer, <img src="http://rudorfer.homedns.org/apl94/img3.gif" alt="$g(\cdot)$" width="29" height="31" align="MIDDLE" border="0" /> is the identity function. The activation level of one layer is propagated to the next layer of the network. Then the weights between the neurons are changed by the backpropagation learning rule. The artificial neural network learns the input/output mapping by a stepwise change of the weights and minimizes the difference between the actual and desired output vector.</p>
<p>The simulation can be divided into two main phases during network training: A randomly selected input/output pair is presented to the input layer of the network. The activation is then propagated to the hidden layers and finally to the output layer of the network.</p>
<p>In the next step the actual output vector is compared with the desired result. Error values are assigned to each neuron in the output layer. The error values are propagated back from the output layer to the hidden layers. The weights are changed so that there is a lower error for a new presentation of the same pattern. The so called ``generalized delta rule'' is used as learning procedure in multi layered perceptron networks.</p>
<p>The weight change in layer <em>l</em> at time <em>v</em> is calculated by</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img9.gif" alt="\begin{displaymath} \Delta w_{ij}^{(l)}(v) = \eta \delta_i^{(l)} x_j^{(l-1)} + \alpha \Delta w_{ij}^{(l)}(v-1)\end{displaymath}" width="280" height="31" /></p>
<p>where <img src="http://rudorfer.homedns.org/apl94/img10.gif" alt="$\eta\in(0,1)$" width="69" height="31" align="MIDDLE" border="0" /> is the learning rate and <img src="http://rudorfer.homedns.org/apl94/img11.gif" alt="$\alpha\in(0,1)$" width="71" height="31" align="MIDDLE" border="0" /> is the momentum. Both are kept constant during learning. <img src="http://rudorfer.homedns.org/apl94/img12.gif" alt="$\delta_i^{(l)}$" width="27" height="40" align="MIDDLE" border="0" /> is defined</p>
<dl compact="compact"><dt>1.</dt><dd>for the output layer (l = L) <img src="http://rudorfer.homedns.org/apl94/img13.gif" alt="$\delta_i^{(L)}$" width="32" height="40" align="MIDDLE" border="0" /> as
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img14.gif" alt="\begin{displaymath} \delta_i^{(L)} = (d_i - x_i^{(L)}) g'(u_i^{(L)}) \end{displaymath}" width="182" height="29" /></p>
where <em>g</em>'(<em>u</em><sub><em>i</em></sub><sup>(<em>L</em>)</sup>) is the gradient of the output function at <em>u</em><sub><em>i</em></sub><sup>(<em>L</em>)</sup>.The gradient of the output function is always positive. <br />
<div align="CENTER"><a name="291"> </a>
<table><caption><strong>Figure 2:</strong> <a name="gener_delta.eps"> </a> Weight adaptation between two neurons</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img15.gif" alt="\begin{figure} \epsfxsize=5cm \centerline{ \epsfbox {gener_delta.eps} } \end{figure}" width="229" height="182" /></td>
</tr>
</tbody>
</table>
</div>
<br />
<p>The formula can be explained as follows: When the output <em>x</em><sub><em>k</em></sub><sup>(<em>l</em>)</sup> of the neuron <em>i</em> in layer <em>l</em> is too small, <img src="http://rudorfer.homedns.org/apl94/img16.gif" alt="$\delta_k^{(l)}$" width="27" height="40" align="MIDDLE" border="0" /> has a negative value. Hence the output of the neuron can be raised by increasing the net input <em>u</em><sub><em>k</em></sub><sup>(<em>l</em>)</sup> by the following change of the weight values:</p>
<p>if <em>x</em><sub><em>i</em></sub><sup>(<em>l</em>-1)</sup> > 0, then increase <em>w</em><sub><em>ki</em></sub><sup>(<em>l</em>)</sup></p>
<p>if <em>x</em><sub><em>i</em></sub><sup>(<em>l</em>-1)</sup> < 0, then decrease <em>w</em><sub><em>ki</em></sub><sup>(<em>l</em>)</sup></p>
<p>The rule applies vice versa for a neuron with an output value that is too high (see figure <a href="http://rudorfer.homedns.org/apl94/node3.html#gener_delta.eps">2</a>).</p>
</dd><dt>2.</dt><dd>for all neurons underneath the output layer (<em>l</em> < <em>L</em>) <img src="http://rudorfer.homedns.org/apl94/img12.gif" alt="$\delta_i^{(l)}$" width="27" height="40" align="MIDDLE" border="0" /> is defined by:
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img17.gif" alt="\begin{displaymath} \delta_i^{(l)} = g'(u_i^{(l)}) \sum_{k=1}^{n^{(l+1)}} \delta_k^{(l+1)} w_{ki}^{(l+1)} \end{displaymath}" width="221" height="61" /></p>
</dd></dl>
<p>Finally the weights of layer <em>l</em> are adjusted by</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img18.gif" alt="\begin{displaymath} w_{ij}^{(l,{\rm new})} = w_{ij}^{(l)} + \Delta w_{ij}^{(l)}(v) \end{displaymath}" width="187" height="31" /></p>
<h1><a name="SECTION00040000000000000000"> Implementation</a></h1>
<p>The time series modeling and forecasting system was implemented using Dyalog APL on HP 9000/700 workstations [<a href="http://rudorfer.homedns.org/apl94/node17.html#dyalogman">Dya91</a>] using the X11 window interface routines provided by the Xfns auxiliary processor. The system consists of two main components:</p>
<ul>
<li>A toolkit of APL functions that drive the neural network and log parameters and results of the simulation runs to APL component files.</li>
<li>An X11-based graphical user interface that allows the user to navigate through the simulations and to compare the actual time series with the one generated by the neural network.</li>
</ul>
<p>Figure <a href="http://rudorfer.homedns.org/apl94/node4.html#fig:screen">3</a> presents a screen dump<a href="http://rudorfer.homedns.org/apl94/footnode.html#294" name="tex2html4"><sup><img src="http://rudorfer.homedns.org/latex2html/foot_motif.gif" alt="[*]" align="BOTTOM" border="1" /></sup></a> of the user interface: the broken line shows the actual time series, the solid line represents the network's output. The forecast data is separated from the historical data - the network's training set - by the vertical bar in the right quarter of the graph. The menu in the upper left corner of figure <a href="http://rudorfer.homedns.org/apl94/node4.html#fig:screen">3</a> allows the user to select a view of the network's forecasting capability at different states throughout the learning phase.</p>
<p>By browsing through the logfiles of the simulation runs, past and present results can be compared and analyzed.</p>
<p> </p>
<p><a name="fig:screen"> </a><a name="138"> </a></p>
<table><caption><strong>Figure 3:</strong> User interface of the forecasting system</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img20.gif" alt="\begin{figure} \epsfxsize=8cm \centerline{ \epsfbox {screen.eps} }\end{figure}" width="378" height="330" /></td>
</tr>
</tbody>
</table>
<h1><a name="SECTION00050000000000000000"> Modeling</a></h1>
<p> </p>
<hr /><!--Table of Child-Links-->
<p><a name="CHILD_LINKS"> </a></p>
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node6.html#SECTION00051000000000000000" name="tex2html87"> The Training Sets</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node7.html#SECTION00052000000000000000" name="tex2html88"> The Algorithm</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node8.html#SECTION00053000000000000000" name="tex2html89"> Network Parameters</a></li>
</ul>
<h2><a name="SECTION00051000000000000000"> The Training Sets</a></h2>
<p>As test bed for our forecasting system we used two well known time series from [<a href="http://rudorfer.homedns.org/apl94/node17.html#box-jenkins">BJ76</a>]: The monthly totals of international airline passengers (thousand of passengers) from 1949 to 1960 (see figure <a href="http://rudorfer.homedns.org/apl94/node6.html#pic:airline_pass">4</a>), and the daily closing prices of IBM common stock from May 1961 to November 1962 (see figure <a href="http://rudorfer.homedns.org/apl94/node6.html#pic:ibm_shares">5</a>).</p>
<p> </p>
<div align="CENTER"><a name="pic:airline_pass"> </a><a name="148"> </a>
<table><caption><strong>Figure 4:</strong> International airline passengers</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img21.gif" alt="\begin{figure} \centerline{ \epsfbox {airlinepass.eps} }\end{figure}" width="336" height="251" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p> </p>
<div align="CENTER"><a name="pic:ibm_shares"> </a><a name="153"> </a>
<table><caption><strong>Figure 5:</strong> IBM share price</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img22.gif" alt="\begin{figure} \centerline{ \epsfbox {ibmshares.eps} }\end{figure}" width="348" height="249" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p>Table <a href="http://rudorfer.homedns.org/apl94/node6.html#tab:time_series_params">1</a> gives some characteristics of these two time series: <img src="http://rudorfer.homedns.org/apl94/img23.gif" alt="$\sigma$" width="13" height="14" align="BOTTOM" border="0" /> is the standard deviation, <img src="http://rudorfer.homedns.org/apl94/img24.gif" alt="$\mu$" width="13" height="27" align="MIDDLE" border="0" /> the mean, and <em>n</em> the number of observations. The airline time series is an example of time series data with a clear trend and multiplicative seasonality, whereas the IBM share price shows a break in the last third of the series and no obvious trend and/or seasonality.</p>
<p> </p>
<div align="CENTER"><a name="tab:time_series_params"> </a><a name="171"> </a>
<table><caption><strong>Figure:</strong> Properties of time series</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img25.gif" alt="\begin{figure} \leavevmode \begin{center} \begin{tabular} {\vert l\vert r\ver... ...ngers & 119.55 & 280.30 & 144 \\ \hline \end{tabular} \end{center}\end{figure}" width="325" height="66" /></td>
</tr>
</tbody>
</table>
</div>
<p><br /> The next section is concerned with the question: How can a neural network learn a time series?</p>
<p> </p>
<h2><a name="SECTION00052000000000000000"> The Algorithm</a></h2>
<p>The neural network sees the time series <img src="http://rudorfer.homedns.org/apl94/img26.gif" alt="$X_1,\ldots,X_n$" width="84" height="28" align="MIDDLE" border="0" /> in the form of many mappings of an input vector to an output value (see figure <a href="http://rudorfer.homedns.org/apl94/node7.html#pic:time">6</a>). This technique was presented by [<a href="http://rudorfer.homedns.org/apl94/node17.html#ch92">CMMR92</a>].</p>
<p>A number of adjoining data points of the time series (the input window <img src="http://rudorfer.homedns.org/apl94/img27.gif" alt="$X_{t-s},X_{t-s+1}\ldots,X_t$" width="152" height="28" align="MIDDLE" border="0" />) are mapped to the interval [0,1] and used as activation levels for the units of the input layer. The size <em>s</em> of the input window corresponds to the number of input units of the neural network. In a forward path, these activation levels are propagated over one hidden layer to one output unit. The error used for the backpropagation learning algorithm is now computed by comparing the value of the output unit with the transformed value of the time series at time <em>t</em>+1. This error is propagated back to the connections between output and hidden layer and to those between hidden and output layer. After all weights have been updated accordingly, one <em>presentation</em> has been completed. Training a neural network with the backpropagation algorithm usually requires that all representations of the input set (called one <em>epoch</em>) are presented many times. In our examples, we used 60 to 138 epochs.</p>
<p>For the learning of time series data, the representations were presented in a randomly manner: As reported by [<a href="http://rudorfer.homedns.org/apl94/node17.html#ch92">CMMR92</a>], choosing a random location for each representation's input window ensures better network performance and avoids local minima.</p>
<p> </p>
<div align="CENTER"><a name="pic:time"> </a><a name="185"> </a>
<table><caption><strong>Figure 6:</strong> Learning a Time Series</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img28.gif" alt="\begin{figure} \centerline{ \epsffile {time.eps} }\end{figure}" width="377" height="557" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p>The next section is concerned with the selection of the right parameters for the learning algorithm and the selection of a suitable topology for the forecasting network.</p>
<h2><a name="SECTION00053000000000000000"> Network Parameters</a></h2>
<p>The following parameters of the artificial neural network were chosen for a closer inspection:</p>
<ul>
<li>The learning rate <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" />
<p><img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> (<img src="http://rudorfer.homedns.org/apl94/img30.gif" alt="$0<\eta<1$" width="73" height="27" align="MIDDLE" border="0" />) is a scaling factor that tells the learning algorithm how strong the weights of the connections should be adjusted for a given error. A higher <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> can be used to speed up the learning process, but if <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> is too high, the algorithm will ``step over'' the optimum weights. The learning rate <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> is constant across presentations.</p>
</li>
<li>The momentum <img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" />
<p>The momentum parameter <img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" /> (<img src="http://rudorfer.homedns.org/apl94/img32.gif" alt="$0<\alpha<1$" width="75" height="27" align="MIDDLE" border="0" />) is another number that affects the gradient descent of the weights: To prevent each connection from following every little change in the solution space immediately, the momentum term is added that keeps the direction of the previous step [<a href="http://rudorfer.homedns.org/apl94/node17.html#hertzkroghpalmer">HKP91</a>], thus avoiding the descent into local minima. The momentum term is constant across presentations.</p>
</li>
<li>The number of input and the number of hidden units (the network topology).
<p>The number of input units determines the number of periods the neural network ``looks into the past'' when predicting the future. The number of input units is equivalent to the size of the input window.</p>
<p>Whereas it has been shown that one hidden layer is sufficient to approximate continuous function [<a href="http://rudorfer.homedns.org/apl94/node17.html#hornik">HSW89</a>], the number of hidden units necessary is ``not known in general [<a href="http://rudorfer.homedns.org/apl94/node17.html#hertzkroghpalmer">HKP91</a>]''. Other approaches for time series analysis with artificial neural networks report working network topologies (number of neurons in the input-hidden-output layer) of 8-8-1, 6-6-1 [<a href="http://rudorfer.homedns.org/apl94/node17.html#ch92">CMMR92</a>], and 5-5-1 [<a href="http://rudorfer.homedns.org/apl94/node17.html#white88">Whi88</a>].</p>
</li>
</ul>
<p>To examine the distribution of these parameters, we conducted a number of experiments: In subsequent runs of the network, these parameters were systematically changed to explore their effect on the network's modeling and forecasting capabilities.</p>
<p>We used the following terms to measure the modeling quality <em>s</em><sub><em>m</em></sub> and forecasting quality <em>s</em><sub><em>f</em></sub> of our system: For a time series <img src="http://rudorfer.homedns.org/apl94/img26.gif" alt="$X_1,\ldots,X_n$" width="84" height="28" align="MIDDLE" border="0" /><a name="eq:sm"> </a></p>
<table style="width: 100%;" align="CENTER">
<tbody>
<tr valign="MIDDLE">
<td> </td>
<td align="CENTER" nowrap="nowrap"><a name="eq:sm"> </a><img src="http://rudorfer.homedns.org/apl94/img33.gif" alt="\begin{displaymath} s_m = \sqrt{\frac{\sum\limits_{i=1}^n(X_i-\hat{X}_i)^2}{n}} \end{displaymath}" width="165" height="66" /></td>
<td align="CENTER">(1)</td>
</tr>
</tbody>
</table>
<p><a name="eq:sf"> </a></p>
<table style="width: 100%;" align="CENTER">
<tbody>
<tr valign="MIDDLE">
<td> </td>
<td align="CENTER" nowrap="nowrap"><a name="eq:sf"> </a><img src="http://rudorfer.homedns.org/apl94/img34.gif" alt="\begin{displaymath} s_f = \sqrt{\frac{\sum\limits_{i=n+1}^{n+1+r}(X_i-\hat{X}_i)^2}{r}}\end{displaymath}" width="181" height="76" /></td>
<td align="CENTER">(2)</td>
</tr>
</tbody>
</table>
<p> </p>
<p>where <img src="http://rudorfer.homedns.org/apl94/img35.gif" alt="$\hat{X}_i$" width="23" height="37" align="MIDDLE" border="0" /> is the estimate of the artificial neural network for period <em>i</em> and <em>r</em> is the number of forecasting periods. The error <em>s</em><sub><em>m</em></sub> (equation <a href="http://rudorfer.homedns.org/apl94/node8.html#eq:sm">1</a>) estimates the capability of the neural network to mimic the known data set, the error <em>s</em><sub><em>f</em></sub> (equation <a href="http://rudorfer.homedns.org/apl94/node8.html#eq:sf">2</a>) judges the networks's forecast capability for a forecast period of length <em>r</em>. In our experiments, we used <em>r</em>=20.</p>
<p>Note: For reasons of clarity, in this section we only present graphics for the IBM share price time series. The graphics for the airline passenger time series are very similar.</p>
<p>The figures <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:eta_alpha_model">7</a> and <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:eta_alpha_fore">8</a> demonstrate the effect of variations of the learning rate <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" />and the momentum <img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" /> on the modeling (figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:eta_alpha_model">7</a>) and forecast (figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:eta_alpha_fore">8</a>) quality: both graphics give evidence for the robustness of the backpropagation algorithm, high values of both <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> and <img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" /> should be avoided.</p>
<p> </p>
<div align="CENTER"><a name="pic:eta_alpha_model"> </a><a name="302"> </a>
<table><caption><strong>Figure:</strong> Learning rate and momentum, IBM share price, modeling quality</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img36.gif" alt="\begin{figure} \epsfxsize=80mm\epsfysize=80mm \centerline{ \epsfbox [0 247 595 842]{excel/reibmeaw.eps} }\end{figure}" width="319" height="331" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<div align="CENTER"><a name="pic:eta_alpha_fore"> </a><a name="304"> </a>
<table><caption><strong>Figure:</strong> Learning rate and momentum, IBM share price, forecasting quality</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img37.gif" alt="\begin{figure} \epsfxsize=80mm\epsfysize=80mm \centerline{ \epsffile [0 247 595 842]{excel/reibmeap.eps} }\end{figure}" width="319" height="331" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p>The figures <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:inp_hid_model">10</a> and <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:inp_hid_fore">11</a> present the effect of different network topologies on the modeling (figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:inp_hid_model">10</a>) and forecasting (figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:inp_hid_fore">11</a>) quality: The number of input units and the number of hidden units open an interesting view: artificial neural networks with more than approx. 50 hidden units are not suited for the task of time series forecasting. This tendency of ``over-elaborate networks capable of data-miming'' is also reported by [<a href="http://rudorfer.homedns.org/apl94/node17.html#white88">Whi88</a>].</p>
<p>Another parameter we have to consider is the number of presentations. A longer training period does not necessarily result in a better forecasting capability. Figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:overlearn">9</a> demonstrates this ``overlearning'' effect for the IBM share price time series: with an increasing number of presentations, the network memorizes details of the time series data instead of learning its essential features. This loss of generalization power has a negative effect on the network's forecasting ability.</p>
<div align="CENTER"><a name="pic:overlearn"> </a><a name="236"> </a>
<table><caption><strong>Figure 9:</strong> Modeling vs. forecasting ability</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img38.gif" alt="\begin{figure} \centerline{ \epsffile {generalization_ibm.eps} }\end{figure}" width="350" height="252" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p>These estimations of the network's most important parameters, although rough, allowed us to choose reasonable parameters for our performance comparison with the ARIMA technique, described in the next section.</p>
<p> </p>
<div align="CENTER"><a name="pic:inp_hid_model"> </a><a name="307"> </a>
<table><caption><strong>Figure:</strong> Number of input and hidden units, IBM share price, modeling quality</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img39.gif" alt="\begin{figure} \epsfxsize=80mm\epsfysize=80mm \centerline{ \epsffile [0 247 595 842]{excel/reibmihw.eps} }\end{figure}" width="323" height="349" /></td>
</tr>
</tbody>
</table>
</div>
<p><a name="pic:inp_hid_fore"> </a><a name="309"> </a></p>
<table><caption><strong>Figure:</strong> Number of input and hidden units, IBM share price, forecasting quality</caption>
<tbody>
<tr>
<td>
<p><img src="http://rudorfer.homedns.org/apl94/img40.gif" alt="\begin{figure} \epsfxsize=80mm\epsfysize=80mm \centerline{ \epsffile [0 247 595 842]{excel/reibmihp.eps} }\end{figure}" width="322" height="348" /></p>
<h1><a name="SECTION00060000000000000000"> Comparison with ARIMA Modeling</a></h1>
<p><a name="chap:arima"> </a>We compared our results with the results of the ARIMA procedure of the SAS software, an integrated system for data access, management, analysis and presentation. The implementation of the ARIMA procedure of SAS follows the programs described by Box and Jenkins in Part V of their classic [<a href="http://rudorfer.homedns.org/apl94/node17.html#box-jenkins">BJ76</a>].</p>
<p>The ARIMA model is called an autoregressive integrated moving average process of order (<em>p</em>, <em>d</em>, <em>q</em>). It is described by the equation</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img41.gif" alt="\begin{displaymath} a(z) \nabla^d X_t = b(z) U_t \end{displaymath}" width="137" height="28" /></p>
where <em>X</em><sub><em>t</em></sub> stands for in time ordered values of a time series, <img src="http://rudorfer.homedns.org/apl94/img42.gif" alt="$t=1, \ldots,n$" width="87" height="27" align="MIDDLE" border="0" /> for <em>n</em> observations. <em>U</em><sub><em>t</em></sub> is a sequence of random values called ``white noise'' process. The backward difference operator <img src="http://rudorfer.homedns.org/apl94/img19.gif" alt="$\nabla$" width="17" height="15" align="BOTTOM" border="0" /> is defined as
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img43.gif" alt="\begin{displaymath} \nabla X_t = X_t - X_{t-1} = (1 - z)X_t\end{displaymath}" width="222" height="28" /></p>
The variable <em>d</em> states how often the difference should be calculated, <em>z</em> is the so called backward shift operator which is defined as <em>z</em><sup><em>m</em></sup> <em>X</em><sub><em>t</em></sub> = <em>X</em><sub><em>t</em>-<em>m</em></sub>. The autoregressive operator <em>a</em>(<em>z</em>) of order <em>p</em> is defined as
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img44.gif" alt="\begin{displaymath} a(z) = 1 - a_1 z - a_2 z^2 - \ldots - a_p z^p\end{displaymath}" width="249" height="29" /></p>
the moving average operator <em>b</em>(<em>z</em>) of order <em>q</em> is defined as
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img45.gif" alt="\begin{displaymath} b(z) = 1 - b_1 z - b_2 z^2 - \ldots - b_q z^q\end{displaymath}" width="241" height="29" /></p>
<p>We fitted an ARIMA model for each time series using the SAS system and let it predict the next 20 observations of the time series. The last 20 observations were dropped from the time series and used to calculate the prediction error of the models.</p>
<p>The following ARIMA models were calculated for the airline passenger time series (after a logarithmic transformation):</p>
<p align="CENTER">(1-<em>z</em>)(1-<em>z<sup>12</sup></em>)<em>X</em><sub><em>t</em></sub> = (1 - 0.24169<em>z</em> - 0.47962<em>z<sup>12</sup></em>) <em>U</em><sub><em>t</em></sub></p>
and for the IBM time series:
<p align="CENTER">(1-<em>z</em>) <em>X</em><sub><em>t</em></sub> = (1 - 0.10538<em>z</em>) <em>U</em><sub><em>t</em></sub></p>
<p>As an opponent for the ARIMA modeling technique, we selected those networks that delivered the smallest forecast error <em>s</em><sub><em>f</em></sub> for the respective time series data:</p>
<div align="CENTER"><a name="309"> </a>
<table border="1" cellpadding="3"><caption><strong>Figure:</strong> Number of input and hidden units, IBM share price, forecasting quality</caption>
<tbody>
<tr valign="TOP">
<td align="LEFT" nowrap="nowrap">series</td>
<td align="RIGHT" nowrap="nowrap"><img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /></td>
<td align="RIGHT" nowrap="nowrap"><img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" /></td>
<td align="RIGHT" nowrap="nowrap"># input</td>
<td align="RIGHT" nowrap="nowrap"># hidden</td>
</tr>
<tr valign="TOP">
<td align="LEFT" nowrap="nowrap"> </td>
<td align="RIGHT" nowrap="nowrap"> </td>
<td align="RIGHT" nowrap="nowrap"> </td>
<td align="RIGHT" nowrap="nowrap">units</td>
<td align="RIGHT" nowrap="nowrap">units</td>
</tr>
<tr valign="TOP">
<td align="LEFT" nowrap="nowrap">airline</td>
<td align="RIGHT" nowrap="nowrap">0.1</td>
<td align="RIGHT" nowrap="nowrap">0.9</td>
<td align="RIGHT" nowrap="nowrap">70</td>
<td align="RIGHT" nowrap="nowrap">45</td>
</tr>
<tr valign="TOP">
<td align="LEFT" nowrap="nowrap">IBM</td>
<td align="RIGHT" nowrap="nowrap">0.1</td>
<td align="RIGHT" nowrap="nowrap">0.9</td>
<td align="RIGHT" nowrap="nowrap">80</td>
<td align="RIGHT" nowrap="nowrap">30</td>
</tr>
</tbody>
</table>
</div>
<p>In Table <a href="http://rudorfer.homedns.org/apl94/node9.html#tab:vergleich">2</a> the prediction errors for the artificial neural network (ANN), the artificial neural network using the logarithmic and <img src="http://rudorfer.homedns.org/apl94/img19.gif" alt="$\nabla$" width="17" height="15" align="BOTTOM" border="0" /> transformation (ANN log,<img src="http://rudorfer.homedns.org/apl94/img19.gif" alt="$\nabla$" width="17" height="15" align="BOTTOM" border="0" />) and the ARIMA model are compared: The artificial neural network using the logarithmic and <img src="http://rudorfer.homedns.org/apl94/img19.gif" alt="$\nabla$" width="17" height="15" align="BOTTOM" border="0" /> transformed time series outperformed the ARIMA models for both time series, whereas the ``simple'' artificial neural network predicted more accurately only for the IBM shares time series. This behavior can be explained as follows: the larger data range of the airline passenger time series leads to a loss of precision for the untransformed input set. Differencing and logarithmic transformations helped to eliminate the trend and mapped the time series data into a smaller range.</p>
<p> </p>
<a name="tab:vergleich"> </a><a name="270"> </a>
<table><caption><strong>Figure:</strong> Forecasting errors for ANN and ARIMA model</caption>
<tbody>
<tr>
<td>
<p><img src="http://rudorfer.homedns.org/apl94/img46.gif" alt="\begin{figure} \leavevmode \begin{center} \begin{tabular} {\vert l\vert r\ver... ...e price & 7.97 & 7.70 & 11.35 \\ \hline \end{tabular} \end{center}\end{figure}" width="386" height="86" /></p>
<h1><a name="SECTION00070000000000000000"> Dyalog APL ANN Code</a></h1>
<a name="chap:code"> </a> These functions present a complete implementation of the definitions given in the introduction. The main function NET learns the well known XOR (exclusive or) mapping. For reasons of simplicity, the input-output mapping of the XOR function is coded into the function NET.
<p> </p>
<hr /><!--Table of Child-Links--> <a name="CHILD_LINKS"> </a>
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node11.html#SECTION00071000000000000000" name="tex2html138"> BACKWARD</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node12.html#SECTION00072000000000000000" name="tex2html139"> FORWARD</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node13.html#SECTION00073000000000000000" name="tex2html140"> GRADIENT</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node14.html#SECTION00074000000000000000" name="tex2html141"> NET</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node15.html#SECTION00075000000000000000" name="tex2html142"> SQUASH</a></li>
</ul>
<h2><a name="SECTION00071000000000000000"> BACKWARD</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img47.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLZ \APLleftarro... ...PLN\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="340" height="326" align="BOTTOM" border="0" />
<h2><a name="SECTION00072000000000000000"> FORWARD</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img48.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLZ \APLleftarro... ...PLN\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="342" height="189" align="BOTTOM" border="0" />
<h2><a name="SECTION00073000000000000000"> GRADIENT</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img49.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLZ \APLleftarro... ...PLX\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="328" height="104" align="BOTTOM" border="0" />
<h2><a name="SECTION00074000000000000000"> NET</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img50.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLN\APLE\APLT \A... ...PLN\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="349" height="461" align="BOTTOM" border="0" />
<h2><a name="SECTION00075000000000000000"> SQUASH</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img51.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLZ \APLleftarro... ...PLA\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="172" height="58" align="BOTTOM" border="0" /> <br />
<h1><a name="SECTION00080000000000000000">Conclusion and Further Work</a></h1>
<p>We have presented a forecasting system for univariate time series that uses artificial neural networks. This computing devices proved themselves to be viable alternatives to conventional techniques. The system can be used in conjunction with other techniques for time series analysis or as a stand-alone tool.</p>
<p>Further work will include the comparison with other time series analysis techniques, development of hybrid techniques that combine the strength of conventional approaches with artificial neural networks and the application of our system to multivariate time series.</p>
<h2><a name="SECTIONREF">References</a></h2>
<dl compact="compact"><dt><a name="alf91"><strong>Alf91</strong></a></dt><dd>M. Alfonseca. <br />Advanced applications of APL: logic programming, neural networks and hypertext. <br /><em>IBM Systems Journal</em>, 30(4):543-553, 1991.</dd><dt><a name="box-jenkins"><strong>BJ76</strong></a></dt><dd>George E. P. Box and Gwilym M. Jenkins. <br /><em>Time Series Analysis - forecasting and control</em>. <br />Series in Time Series Analysis. Holden-Day, 500 Sansome Street, San Franciso, California, 1976.</dd><dt><a name="chatfield"><strong>Cha91</strong></a></dt><dd>E. Chatfield. <br /><em>The Analysis of Time Series</em>. <br />Chapman and Hall, New York, fourth edition, 1991.</dd><dt><a name="ch92"><strong>CMMR92</strong></a></dt><dd>Kanad Chakraborty, Kishan Mehrota, Chilukuri K. Mohan, and Sanjay Ranka. <br />Forecasting the Behaviour of Multivariate Time Series Using Neural Networds. <br /><em>Neural Networks</em>, 5:961-970, 1992.</dd><dt><a name="dyalogman"><strong>Dya91</strong></a></dt><dd>Dyadic Systems Limited, Riverside View, Basing Road, Old Basing, Basingstoke, Hampshire RG24 0AL, England. <br /><em>Dyalog Apl Users Guide</em>, 1991.</dd><dt><a name="evans91"><strong>ES91</strong></a></dt><dd>Richard M. Evans and Alvin J. Surkan. <br />Relating Numbers of Processing Elements in a Sparse Distributed Memory Model to Learning Rate and Generalization. <br /><em>ACM APL Quote Quad</em>, 21(4):166-173, 1991.</dd><dt><a name="hertzkroghpalmer"><strong>HKP91</strong></a></dt><dd>John Hertz, Anders Krogh, and Richard G. Palmer. <br /><em>Introduction to the Theory od Neural Computation</em>. <br />Addison Wesley, Redwood City, California, 1991.</dd><dt><a name="hornik"><strong>HSW89</strong></a></dt><dd>Kurt Hornik, Maxwell Stinchcombe, and Halbert White. <br />Multilayer Feedforward Networks are Universal Approximators. <br /><em>Neural Networks</em>, 2:359-366, 1989.</dd><dt><a name="peele81"><strong>Pee81</strong></a></dt><dd>Howard A. Peele. <br />Teaching A Topic in Cybernetics with APL: An Introduction to Neural Net Modelling. <br /><em>ACM APL Quote Quad</em>, 12(1):235-239, 1981.</dd><dt><a name="rummel"><strong>RHW86</strong></a></dt><dd>David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. <br />Learning representations by back-propagating errors. <br /><em>Nature</em>, 323(9):533-536, October 1986.</dd><dt><a name="turingnn"><strong>SS91</strong></a></dt><dd>Hava Siegelmann and Eduardo D. Sontag. <br />Neural Nets Are Universal Computing Devices. <br />Technical Report SYSCON-91-08, Rutgers Center for Systems and Control, May 1991.</dd><dt><a name="sku93"><strong>SS93</strong></a></dt><dd>Alexei N. Skurihin and Alvin J. Surkan. <br />Identification of Parallelism in Neural Networks by Simulation with Language J. <br /><em>ACM APL Quote Quad</em>, 24(1):230-237, 1993.</dd><dt><a name="white88"><strong>Whi88</strong></a></dt><dd>Halbert White. <br />Economic prediction using neural networks: the case of ibm daily stock returns. <br />In <em>Proceedings of the IEEE International Conference on Neural Networks</em>, pages II-451-II-459, 1988.</dd></dl><hr /><address> </address>
<table>
<tbody>
<tr>
<td> </td>
<td><em>© 2010 <a href="http://www.ai.wu.ac.at/mitarbeiter/rudorfer.html">Gottfried Rudorfer</a>, © 1994 ACM APL Quote Quad, 1515 Broadway, New York, N.Y. 10036, <a href="http://wwwai.wu-wien.ac.at/">Abteilung für Angewandte Informatik</a>, <a href="http://www.wu-wien.ac.at/">Wirtschaftsuniversität Wien</a>, 3/23/1998</em></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<!--End of Table of Child-Links-->
<p style="text-align: center;"><strong><span style="font-size: 14pt;"><em>Time Series Forecasting Using Neural Networks</em></span></strong></p>
<p align="CENTER"><strong><em><big>Thomas Kolarik and Gottfried Rudorfer <br /> <em> <big><span style="font-size: 10pt;">Department of Applied Computer Science </span><br /> <em> <big><span style="font-size: 10pt;">Vienna University of Economics and Business Administration </span><br /> </big></em></big></em></big><em><em>Augasse 2-6, A-1090 Vienna, Austria<big> <big> <big> <big> <br /> </big></big></big></big><tt><a href="mailto:gottfried@rudorfer.homedns.org?subject=Time%20Series%20Forecasting%20Using%20Neural%20Networks">gottfried@rudorfer.homedns.org</a></tt></em></em></em></strong></p>
<hr />
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node1.html#SECTION00010000000000000000" name="tex2html19"> Abstract</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node2.html#SECTION00020000000000000000" name="tex2html20"> Motivation</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node3.html#SECTION00030000000000000000" name="tex2html21"> A Very Short Introduction to Artificial Neural Networks</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node4.html#SECTION00040000000000000000" name="tex2html22"> Implementation</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node5.html#SECTION00050000000000000000" name="tex2html23"> Modeling</a>
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node6.html#SECTION00051000000000000000" name="tex2html24"> The Training Sets</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node7.html#SECTION00052000000000000000" name="tex2html25"> The Algorithm</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node8.html#SECTION00053000000000000000" name="tex2html26"> Network Parameters</a></li>
</ul>
</li>
<li><a href="http://rudorfer.homedns.org/apl94/node9.html#SECTION00060000000000000000" name="tex2html27"> Comparison with ARIMA Modeling</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node10.html#SECTION00070000000000000000" name="tex2html28"> Dyalog APL ANN Code</a>
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node11.html#SECTION00071000000000000000" name="tex2html29"> BACKWARD</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node12.html#SECTION00072000000000000000" name="tex2html30"> FORWARD</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node13.html#SECTION00073000000000000000" name="tex2html31"> GRADIENT</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node14.html#SECTION00074000000000000000" name="tex2html32"> NET</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node15.html#SECTION00075000000000000000" name="tex2html33"> SQUASH</a></li>
</ul>
</li>
<li><a href="http://rudorfer.homedns.org/apl94/node16.html#SECTION00080000000000000000" name="tex2html34"> Conclusion and Further Work</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node17.html#SECTION00090000000000000000" name="tex2html35"> References</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node18.html#SECTION000100000000000000000" name="tex2html36"> About this document ... </a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node18.html#SECTION000100000000000000000" name="tex2html36"></a></li>
</ul>
<h1><a name="SECTION00010000000000000000"> Abstract</a></h1>
<p>Artificial neural networks are suitable for many tasks in pattern recognition and machine learning. In this paper we present an APL system for forecasting univariate time series with artificial neural networks. Unlike conventional techniques for time series analysis, an artificial neural network needs little information about the time series data and can be applied to a broad range of problems. However, the problem of network ``tuning'' remains: parameters of the backpropagation algorithm as well as the network topology need to be adjusted for optimal performance. For our application, we conducted experiments to find the right parameters for a forecasting network. The artificial neural networks that were found delivered a better forecasting performance than results obtained by the well known ARIMA technique.</p>
<h1><a name="SECTION00020000000000000000"> Motivation</a></h1>
<p>Time series analysis as described by most textbooks [<a href="http://rudorfer.homedns.org/apl94/node17.html#chatfield">Cha91</a>] relies on explicit descriptive, stochastic, spectral or other models of processes that describe the real world phenomena generating the observed data.</p>
<p>Usually, the parameters of a standard model like the ARIMA technique [<a href="http://rudorfer.homedns.org/apl94/node17.html#box-jenkins">BJ76</a>] are derived from the autocorrelation and frequency spectrum of the time series. Problems with the ARIMA approach arise with time series of increasing variance or when the time series represents nonlinear processes.</p>
<p>The usage of artificial neural networks for time series analysis relies purely on the data that were observed. As multi layer feed forward networks with at least one hidden layer and a sufficient number of hidden units are capable of approximating any measurable function [<a href="http://rudorfer.homedns.org/apl94/node17.html#hornik">HSW89</a>,<a href="http://rudorfer.homedns.org/apl94/node17.html#turingnn">SS91</a>], an artificial neural network is powerful enough to represent any form of time series. The capability to generalize allows artificial neural networks to learn even in the case of noisy and/or missing data. Another advantage over linear models like the ARIMA technique is the network's ability to represent nonlinear time series.</p>
<p>The APL programming language is very suitable for the task of implementing neural networks [<a href="http://rudorfer.homedns.org/apl94/node17.html#alf91">Alf91</a>,<a href="http://rudorfer.homedns.org/apl94/node17.html#peele81">Pee81</a>,<a href="http://rudorfer.homedns.org/apl94/node17.html#evans91">ES91</a>,<a href="http://rudorfer.homedns.org/apl94/node17.html#sku93">SS93</a>] because of its ability of handle matrix and vector operations. The forward and backward paths of a fully connected feed forward network can be implemented by outer and inner products of vectors and matrices in a few lines of APL code.</p>
<p>For our application, we decided to use a fully connected, layered, feed forward artificial neural network with one hidden layer and the backpropagation learning algorithm. The next section gives a short overview of the relevant definitions and algorithms.</p>
<h1><a name="SECTION00030000000000000000"> A Very Short Introduction to Artificial Neural Networks</a></h1>
<p><a name="chap:intro"> </a> As mentioned above, our simulations utilized the ``multi layered perceptron model'' (MLP), also known as ``feed forward networks'' trained with the ``generalized delta rule'', also known as ``backpropagation''.</p>
<p>The foundations of the backpropagation method for learning in neural networks were laid by [<a href="http://rudorfer.homedns.org/apl94/node17.html#rummel">RHW86</a>].</p>
<p>Artificial neural networks consist of many simple processing devices (called processing elements or neurons) grouped in layers. Each layer is identified by the index <img src="http://rudorfer.homedns.org/apl94/img1.gif" alt="$l=0,\ldots,L$" width="87" height="29" align="MIDDLE" border="0" />. The layers and <em>L</em> are called the ``input layer'' and ``output layer'', all other layers are called ``hidden layers''. The processing elements are interconnected as follows: Communication between processing elements is only allowed for processing elements of neighbouring layers. Neurons within a layer cannot communicate. Each neuron has a certain activation level <em>a</em>. The network processes data by the exchange of activation levels between connected neurons (see figure <a href="http://rudorfer.homedns.org/apl94/node3.html#multi_detail">1</a>):</p>
<p> </p>
<div align="CENTER"><a name="285"> </a>
<table><caption><strong>Figure:</strong> <a name="multi_detail"> </a> Exchange of activation values between neurons</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img2.gif" alt="\begin{figure} \centerline{ \epsfbox {multi_detail.eps} } \end{figure}" width="300" height="610" /></td>
</tr>
</tbody>
</table>
</div>
<p><br /> The output value of the <em>i</em>-th neuron in layer <em>l</em> is denoted by <em>x</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup>. It is calculated with the formula</p>
<p align="CENTER"><em>x</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup> = <em>g</em>(<em>a</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup>)</p>
<p>where <img src="http://rudorfer.homedns.org/apl94/img3.gif" alt="$g(\cdot)$" width="29" height="31" align="MIDDLE" border="0" /> is a monotone increasing function. For our examples, we use the function <img src="http://rudorfer.homedns.org/apl94/img4.gif" alt="$g(y)=\frac{1}{1 + e^{-y}}$" width="99" height="34" align="MIDDLE" border="0" /> (the ``squashing function''). The activation level <em>a</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup> of the neuron <em>i</em> in layer <em>l</em> is calculated by</p>
<p align="CENTER"><em>a</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup> = <em>f</em>(<em>u</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup>)</p>
<p>where <img src="http://rudorfer.homedns.org/apl94/img5.gif" alt="$f(\cdot)$" width="31" height="31" align="MIDDLE" border="0" /> is the activation function (in our case the identity function is used).</p>
<p>The net input <em>u</em><sub><em>i</em></sub><sup>(<em>l</em>)</sup> of neuron <em>i</em> in layer <em>l</em> is calculated as</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img6.gif" alt="\begin{displaymath} u_i^{(l)} = \biggl( \sum_{j = 1}^{n^{(l-1)}} {w_{ij}^{(l)} x_j^{(l-1)} \biggr)} - \Theta_i^{(l)}\end{displaymath}" width="228" height="63" /></p>
<p>where <em>w</em><sub><em>ij</em></sub><sup>(<em>l</em>)</sup> is the weight of neuron <em>j</em> in layer <em>l</em>-1 connected to neuron <em>i</em> in layer <em>l</em>, <em>x</em><sub><em>j</em></sub><sup>(<em>l</em>-1)</sup> is the output of neuron <em>j</em> in layer <em>l</em>-1. <img src="http://rudorfer.homedns.org/apl94/img7.gif" alt="$\Theta_i^{(l)}$" width="32" height="40" align="MIDDLE" border="0" /> is a bias value that is subtracted from the sum of the weighted activations.</p>
<p>The calculation of the network status starts at the input layer and ends at the output layer. The input vector <em>I</em> initializes the activation levels of the neurons in the input layer:</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img8.gif" alt="\begin{displaymath} a^{(0)}_i = {\rm i^{\rm th}~element~of~I}\end{displaymath}" width="159" height="29" /></p>
<p>For the input layer, <img src="http://rudorfer.homedns.org/apl94/img3.gif" alt="$g(\cdot)$" width="29" height="31" align="MIDDLE" border="0" /> is the identity function. The activation level of one layer is propagated to the next layer of the network. Then the weights between the neurons are changed by the backpropagation learning rule. The artificial neural network learns the input/output mapping by a stepwise change of the weights and minimizes the difference between the actual and desired output vector.</p>
<p>The simulation can be divided into two main phases during network training: A randomly selected input/output pair is presented to the input layer of the network. The activation is then propagated to the hidden layers and finally to the output layer of the network.</p>
<p>In the next step the actual output vector is compared with the desired result. Error values are assigned to each neuron in the output layer. The error values are propagated back from the output layer to the hidden layers. The weights are changed so that there is a lower error for a new presentation of the same pattern. The so called ``generalized delta rule'' is used as learning procedure in multi layered perceptron networks.</p>
<p>The weight change in layer <em>l</em> at time <em>v</em> is calculated by</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img9.gif" alt="\begin{displaymath} \Delta w_{ij}^{(l)}(v) = \eta \delta_i^{(l)} x_j^{(l-1)} + \alpha \Delta w_{ij}^{(l)}(v-1)\end{displaymath}" width="280" height="31" /></p>
<p>where <img src="http://rudorfer.homedns.org/apl94/img10.gif" alt="$\eta\in(0,1)$" width="69" height="31" align="MIDDLE" border="0" /> is the learning rate and <img src="http://rudorfer.homedns.org/apl94/img11.gif" alt="$\alpha\in(0,1)$" width="71" height="31" align="MIDDLE" border="0" /> is the momentum. Both are kept constant during learning. <img src="http://rudorfer.homedns.org/apl94/img12.gif" alt="$\delta_i^{(l)}$" width="27" height="40" align="MIDDLE" border="0" /> is defined</p>
<dl compact="compact"><dt>1.</dt><dd>for the output layer (l = L) <img src="http://rudorfer.homedns.org/apl94/img13.gif" alt="$\delta_i^{(L)}$" width="32" height="40" align="MIDDLE" border="0" /> as
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img14.gif" alt="\begin{displaymath} \delta_i^{(L)} = (d_i - x_i^{(L)}) g'(u_i^{(L)}) \end{displaymath}" width="182" height="29" /></p>
where <em>g</em>'(<em>u</em><sub><em>i</em></sub><sup>(<em>L</em>)</sup>) is the gradient of the output function at <em>u</em><sub><em>i</em></sub><sup>(<em>L</em>)</sup>.The gradient of the output function is always positive. <br />
<div align="CENTER"><a name="291"> </a>
<table><caption><strong>Figure 2:</strong> <a name="gener_delta.eps"> </a> Weight adaptation between two neurons</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img15.gif" alt="\begin{figure} \epsfxsize=5cm \centerline{ \epsfbox {gener_delta.eps} } \end{figure}" width="229" height="182" /></td>
</tr>
</tbody>
</table>
</div>
<br />
<p>The formula can be explained as follows: When the output <em>x</em><sub><em>k</em></sub><sup>(<em>l</em>)</sup> of the neuron <em>i</em> in layer <em>l</em> is too small, <img src="http://rudorfer.homedns.org/apl94/img16.gif" alt="$\delta_k^{(l)}$" width="27" height="40" align="MIDDLE" border="0" /> has a negative value. Hence the output of the neuron can be raised by increasing the net input <em>u</em><sub><em>k</em></sub><sup>(<em>l</em>)</sup> by the following change of the weight values:</p>
<p>if <em>x</em><sub><em>i</em></sub><sup>(<em>l</em>-1)</sup> > 0, then increase <em>w</em><sub><em>ki</em></sub><sup>(<em>l</em>)</sup></p>
<p>if <em>x</em><sub><em>i</em></sub><sup>(<em>l</em>-1)</sup> < 0, then decrease <em>w</em><sub><em>ki</em></sub><sup>(<em>l</em>)</sup></p>
<p>The rule applies vice versa for a neuron with an output value that is too high (see figure <a href="http://rudorfer.homedns.org/apl94/node3.html#gener_delta.eps">2</a>).</p>
</dd><dt>2.</dt><dd>for all neurons underneath the output layer (<em>l</em> < <em>L</em>) <img src="http://rudorfer.homedns.org/apl94/img12.gif" alt="$\delta_i^{(l)}$" width="27" height="40" align="MIDDLE" border="0" /> is defined by:
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img17.gif" alt="\begin{displaymath} \delta_i^{(l)} = g'(u_i^{(l)}) \sum_{k=1}^{n^{(l+1)}} \delta_k^{(l+1)} w_{ki}^{(l+1)} \end{displaymath}" width="221" height="61" /></p>
</dd></dl>
<p>Finally the weights of layer <em>l</em> are adjusted by</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img18.gif" alt="\begin{displaymath} w_{ij}^{(l,{\rm new})} = w_{ij}^{(l)} + \Delta w_{ij}^{(l)}(v) \end{displaymath}" width="187" height="31" /></p>
<h1><a name="SECTION00040000000000000000"> Implementation</a></h1>
<p>The time series modeling and forecasting system was implemented using Dyalog APL on HP 9000/700 workstations [<a href="http://rudorfer.homedns.org/apl94/node17.html#dyalogman">Dya91</a>] using the X11 window interface routines provided by the Xfns auxiliary processor. The system consists of two main components:</p>
<ul>
<li>A toolkit of APL functions that drive the neural network and log parameters and results of the simulation runs to APL component files.</li>
<li>An X11-based graphical user interface that allows the user to navigate through the simulations and to compare the actual time series with the one generated by the neural network.</li>
</ul>
<p>Figure <a href="http://rudorfer.homedns.org/apl94/node4.html#fig:screen">3</a> presents a screen dump<a href="http://rudorfer.homedns.org/apl94/footnode.html#294" name="tex2html4"><sup><img src="http://rudorfer.homedns.org/latex2html/foot_motif.gif" alt="[*]" align="BOTTOM" border="1" /></sup></a> of the user interface: the broken line shows the actual time series, the solid line represents the network's output. The forecast data is separated from the historical data - the network's training set - by the vertical bar in the right quarter of the graph. The menu in the upper left corner of figure <a href="http://rudorfer.homedns.org/apl94/node4.html#fig:screen">3</a> allows the user to select a view of the network's forecasting capability at different states throughout the learning phase.</p>
<p>By browsing through the logfiles of the simulation runs, past and present results can be compared and analyzed.</p>
<p> </p>
<p><a name="fig:screen"> </a><a name="138"> </a></p>
<table><caption><strong>Figure 3:</strong> User interface of the forecasting system</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img20.gif" alt="\begin{figure} \epsfxsize=8cm \centerline{ \epsfbox {screen.eps} }\end{figure}" width="378" height="330" /></td>
</tr>
</tbody>
</table>
<h1><a name="SECTION00050000000000000000"> Modeling</a></h1>
<p> </p>
<hr /><!--Table of Child-Links-->
<p><a name="CHILD_LINKS"> </a></p>
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node6.html#SECTION00051000000000000000" name="tex2html87"> The Training Sets</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node7.html#SECTION00052000000000000000" name="tex2html88"> The Algorithm</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node8.html#SECTION00053000000000000000" name="tex2html89"> Network Parameters</a></li>
</ul>
<h2><a name="SECTION00051000000000000000"> The Training Sets</a></h2>
<p>As test bed for our forecasting system we used two well known time series from [<a href="http://rudorfer.homedns.org/apl94/node17.html#box-jenkins">BJ76</a>]: The monthly totals of international airline passengers (thousand of passengers) from 1949 to 1960 (see figure <a href="http://rudorfer.homedns.org/apl94/node6.html#pic:airline_pass">4</a>), and the daily closing prices of IBM common stock from May 1961 to November 1962 (see figure <a href="http://rudorfer.homedns.org/apl94/node6.html#pic:ibm_shares">5</a>).</p>
<p> </p>
<div align="CENTER"><a name="pic:airline_pass"> </a><a name="148"> </a>
<table><caption><strong>Figure 4:</strong> International airline passengers</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img21.gif" alt="\begin{figure} \centerline{ \epsfbox {airlinepass.eps} }\end{figure}" width="336" height="251" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p> </p>
<div align="CENTER"><a name="pic:ibm_shares"> </a><a name="153"> </a>
<table><caption><strong>Figure 5:</strong> IBM share price</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img22.gif" alt="\begin{figure} \centerline{ \epsfbox {ibmshares.eps} }\end{figure}" width="348" height="249" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p>Table <a href="http://rudorfer.homedns.org/apl94/node6.html#tab:time_series_params">1</a> gives some characteristics of these two time series: <img src="http://rudorfer.homedns.org/apl94/img23.gif" alt="$\sigma$" width="13" height="14" align="BOTTOM" border="0" /> is the standard deviation, <img src="http://rudorfer.homedns.org/apl94/img24.gif" alt="$\mu$" width="13" height="27" align="MIDDLE" border="0" /> the mean, and <em>n</em> the number of observations. The airline time series is an example of time series data with a clear trend and multiplicative seasonality, whereas the IBM share price shows a break in the last third of the series and no obvious trend and/or seasonality.</p>
<p> </p>
<div align="CENTER"><a name="tab:time_series_params"> </a><a name="171"> </a>
<table><caption><strong>Figure:</strong> Properties of time series</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img25.gif" alt="\begin{figure} \leavevmode \begin{center} \begin{tabular} {\vert l\vert r\ver... ...ngers & 119.55 & 280.30 & 144 \\ \hline \end{tabular} \end{center}\end{figure}" width="325" height="66" /></td>
</tr>
</tbody>
</table>
</div>
<p><br /> The next section is concerned with the question: How can a neural network learn a time series?</p>
<p> </p>
<h2><a name="SECTION00052000000000000000"> The Algorithm</a></h2>
<p>The neural network sees the time series <img src="http://rudorfer.homedns.org/apl94/img26.gif" alt="$X_1,\ldots,X_n$" width="84" height="28" align="MIDDLE" border="0" /> in the form of many mappings of an input vector to an output value (see figure <a href="http://rudorfer.homedns.org/apl94/node7.html#pic:time">6</a>). This technique was presented by [<a href="http://rudorfer.homedns.org/apl94/node17.html#ch92">CMMR92</a>].</p>
<p>A number of adjoining data points of the time series (the input window <img src="http://rudorfer.homedns.org/apl94/img27.gif" alt="$X_{t-s},X_{t-s+1}\ldots,X_t$" width="152" height="28" align="MIDDLE" border="0" />) are mapped to the interval [0,1] and used as activation levels for the units of the input layer. The size <em>s</em> of the input window corresponds to the number of input units of the neural network. In a forward path, these activation levels are propagated over one hidden layer to one output unit. The error used for the backpropagation learning algorithm is now computed by comparing the value of the output unit with the transformed value of the time series at time <em>t</em>+1. This error is propagated back to the connections between output and hidden layer and to those between hidden and output layer. After all weights have been updated accordingly, one <em>presentation</em> has been completed. Training a neural network with the backpropagation algorithm usually requires that all representations of the input set (called one <em>epoch</em>) are presented many times. In our examples, we used 60 to 138 epochs.</p>
<p>For the learning of time series data, the representations were presented in a randomly manner: As reported by [<a href="http://rudorfer.homedns.org/apl94/node17.html#ch92">CMMR92</a>], choosing a random location for each representation's input window ensures better network performance and avoids local minima.</p>
<p> </p>
<div align="CENTER"><a name="pic:time"> </a><a name="185"> </a>
<table><caption><strong>Figure 6:</strong> Learning a Time Series</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img28.gif" alt="\begin{figure} \centerline{ \epsffile {time.eps} }\end{figure}" width="377" height="557" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p>The next section is concerned with the selection of the right parameters for the learning algorithm and the selection of a suitable topology for the forecasting network.</p>
<h2><a name="SECTION00053000000000000000"> Network Parameters</a></h2>
<p>The following parameters of the artificial neural network were chosen for a closer inspection:</p>
<ul>
<li>The learning rate <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" />
<p><img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> (<img src="http://rudorfer.homedns.org/apl94/img30.gif" alt="$0<\eta<1$" width="73" height="27" align="MIDDLE" border="0" />) is a scaling factor that tells the learning algorithm how strong the weights of the connections should be adjusted for a given error. A higher <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> can be used to speed up the learning process, but if <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> is too high, the algorithm will ``step over'' the optimum weights. The learning rate <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> is constant across presentations.</p>
</li>
<li>The momentum <img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" />
<p>The momentum parameter <img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" /> (<img src="http://rudorfer.homedns.org/apl94/img32.gif" alt="$0<\alpha<1$" width="75" height="27" align="MIDDLE" border="0" />) is another number that affects the gradient descent of the weights: To prevent each connection from following every little change in the solution space immediately, the momentum term is added that keeps the direction of the previous step [<a href="http://rudorfer.homedns.org/apl94/node17.html#hertzkroghpalmer">HKP91</a>], thus avoiding the descent into local minima. The momentum term is constant across presentations.</p>
</li>
<li>The number of input and the number of hidden units (the network topology).
<p>The number of input units determines the number of periods the neural network ``looks into the past'' when predicting the future. The number of input units is equivalent to the size of the input window.</p>
<p>Whereas it has been shown that one hidden layer is sufficient to approximate continuous function [<a href="http://rudorfer.homedns.org/apl94/node17.html#hornik">HSW89</a>], the number of hidden units necessary is ``not known in general [<a href="http://rudorfer.homedns.org/apl94/node17.html#hertzkroghpalmer">HKP91</a>]''. Other approaches for time series analysis with artificial neural networks report working network topologies (number of neurons in the input-hidden-output layer) of 8-8-1, 6-6-1 [<a href="http://rudorfer.homedns.org/apl94/node17.html#ch92">CMMR92</a>], and 5-5-1 [<a href="http://rudorfer.homedns.org/apl94/node17.html#white88">Whi88</a>].</p>
</li>
</ul>
<p>To examine the distribution of these parameters, we conducted a number of experiments: In subsequent runs of the network, these parameters were systematically changed to explore their effect on the network's modeling and forecasting capabilities.</p>
<p>We used the following terms to measure the modeling quality <em>s</em><sub><em>m</em></sub> and forecasting quality <em>s</em><sub><em>f</em></sub> of our system: For a time series <img src="http://rudorfer.homedns.org/apl94/img26.gif" alt="$X_1,\ldots,X_n$" width="84" height="28" align="MIDDLE" border="0" /><a name="eq:sm"> </a></p>
<table style="width: 100%;" align="CENTER">
<tbody>
<tr valign="MIDDLE">
<td> </td>
<td align="CENTER" nowrap="nowrap"><a name="eq:sm"> </a><img src="http://rudorfer.homedns.org/apl94/img33.gif" alt="\begin{displaymath} s_m = \sqrt{\frac{\sum\limits_{i=1}^n(X_i-\hat{X}_i)^2}{n}} \end{displaymath}" width="165" height="66" /></td>
<td align="CENTER">(1)</td>
</tr>
</tbody>
</table>
<p><a name="eq:sf"> </a></p>
<table style="width: 100%;" align="CENTER">
<tbody>
<tr valign="MIDDLE">
<td> </td>
<td align="CENTER" nowrap="nowrap"><a name="eq:sf"> </a><img src="http://rudorfer.homedns.org/apl94/img34.gif" alt="\begin{displaymath} s_f = \sqrt{\frac{\sum\limits_{i=n+1}^{n+1+r}(X_i-\hat{X}_i)^2}{r}}\end{displaymath}" width="181" height="76" /></td>
<td align="CENTER">(2)</td>
</tr>
</tbody>
</table>
<p> </p>
<p>where <img src="http://rudorfer.homedns.org/apl94/img35.gif" alt="$\hat{X}_i$" width="23" height="37" align="MIDDLE" border="0" /> is the estimate of the artificial neural network for period <em>i</em> and <em>r</em> is the number of forecasting periods. The error <em>s</em><sub><em>m</em></sub> (equation <a href="http://rudorfer.homedns.org/apl94/node8.html#eq:sm">1</a>) estimates the capability of the neural network to mimic the known data set, the error <em>s</em><sub><em>f</em></sub> (equation <a href="http://rudorfer.homedns.org/apl94/node8.html#eq:sf">2</a>) judges the networks's forecast capability for a forecast period of length <em>r</em>. In our experiments, we used <em>r</em>=20.</p>
<p>Note: For reasons of clarity, in this section we only present graphics for the IBM share price time series. The graphics for the airline passenger time series are very similar.</p>
<p>The figures <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:eta_alpha_model">7</a> and <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:eta_alpha_fore">8</a> demonstrate the effect of variations of the learning rate <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" />and the momentum <img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" /> on the modeling (figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:eta_alpha_model">7</a>) and forecast (figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:eta_alpha_fore">8</a>) quality: both graphics give evidence for the robustness of the backpropagation algorithm, high values of both <img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /> and <img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" /> should be avoided.</p>
<p> </p>
<div align="CENTER"><a name="pic:eta_alpha_model"> </a><a name="302"> </a>
<table><caption><strong>Figure:</strong> Learning rate and momentum, IBM share price, modeling quality</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img36.gif" alt="\begin{figure} \epsfxsize=80mm\epsfysize=80mm \centerline{ \epsfbox [0 247 595 842]{excel/reibmeaw.eps} }\end{figure}" width="319" height="331" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<div align="CENTER"><a name="pic:eta_alpha_fore"> </a><a name="304"> </a>
<table><caption><strong>Figure:</strong> Learning rate and momentum, IBM share price, forecasting quality</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img37.gif" alt="\begin{figure} \epsfxsize=80mm\epsfysize=80mm \centerline{ \epsffile [0 247 595 842]{excel/reibmeap.eps} }\end{figure}" width="319" height="331" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p>The figures <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:inp_hid_model">10</a> and <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:inp_hid_fore">11</a> present the effect of different network topologies on the modeling (figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:inp_hid_model">10</a>) and forecasting (figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:inp_hid_fore">11</a>) quality: The number of input units and the number of hidden units open an interesting view: artificial neural networks with more than approx. 50 hidden units are not suited for the task of time series forecasting. This tendency of ``over-elaborate networks capable of data-miming'' is also reported by [<a href="http://rudorfer.homedns.org/apl94/node17.html#white88">Whi88</a>].</p>
<p>Another parameter we have to consider is the number of presentations. A longer training period does not necessarily result in a better forecasting capability. Figure <a href="http://rudorfer.homedns.org/apl94/node8.html#pic:overlearn">9</a> demonstrates this ``overlearning'' effect for the IBM share price time series: with an increasing number of presentations, the network memorizes details of the time series data instead of learning its essential features. This loss of generalization power has a negative effect on the network's forecasting ability.</p>
<div align="CENTER"><a name="pic:overlearn"> </a><a name="236"> </a>
<table><caption><strong>Figure 9:</strong> Modeling vs. forecasting ability</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img38.gif" alt="\begin{figure} \centerline{ \epsffile {generalization_ibm.eps} }\end{figure}" width="350" height="252" /></td>
</tr>
</tbody>
</table>
</div>
<p> </p>
<p>These estimations of the network's most important parameters, although rough, allowed us to choose reasonable parameters for our performance comparison with the ARIMA technique, described in the next section.</p>
<p> </p>
<div align="CENTER"><a name="pic:inp_hid_model"> </a><a name="307"> </a>
<table><caption><strong>Figure:</strong> Number of input and hidden units, IBM share price, modeling quality</caption>
<tbody>
<tr>
<td><img src="http://rudorfer.homedns.org/apl94/img39.gif" alt="\begin{figure} \epsfxsize=80mm\epsfysize=80mm \centerline{ \epsffile [0 247 595 842]{excel/reibmihw.eps} }\end{figure}" width="323" height="349" /></td>
</tr>
</tbody>
</table>
</div>
<p><a name="pic:inp_hid_fore"> </a><a name="309"> </a></p>
<table><caption><strong>Figure:</strong> Number of input and hidden units, IBM share price, forecasting quality</caption>
<tbody>
<tr>
<td>
<p><img src="http://rudorfer.homedns.org/apl94/img40.gif" alt="\begin{figure} \epsfxsize=80mm\epsfysize=80mm \centerline{ \epsffile [0 247 595 842]{excel/reibmihp.eps} }\end{figure}" width="322" height="348" /></p>
<h1><a name="SECTION00060000000000000000"> Comparison with ARIMA Modeling</a></h1>
<p><a name="chap:arima"> </a>We compared our results with the results of the ARIMA procedure of the SAS software, an integrated system for data access, management, analysis and presentation. The implementation of the ARIMA procedure of SAS follows the programs described by Box and Jenkins in Part V of their classic [<a href="http://rudorfer.homedns.org/apl94/node17.html#box-jenkins">BJ76</a>].</p>
<p>The ARIMA model is called an autoregressive integrated moving average process of order (<em>p</em>, <em>d</em>, <em>q</em>). It is described by the equation</p>
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img41.gif" alt="\begin{displaymath} a(z) \nabla^d X_t = b(z) U_t \end{displaymath}" width="137" height="28" /></p>
where <em>X</em><sub><em>t</em></sub> stands for in time ordered values of a time series, <img src="http://rudorfer.homedns.org/apl94/img42.gif" alt="$t=1, \ldots,n$" width="87" height="27" align="MIDDLE" border="0" /> for <em>n</em> observations. <em>U</em><sub><em>t</em></sub> is a sequence of random values called ``white noise'' process. The backward difference operator <img src="http://rudorfer.homedns.org/apl94/img19.gif" alt="$\nabla$" width="17" height="15" align="BOTTOM" border="0" /> is defined as
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img43.gif" alt="\begin{displaymath} \nabla X_t = X_t - X_{t-1} = (1 - z)X_t\end{displaymath}" width="222" height="28" /></p>
The variable <em>d</em> states how often the difference should be calculated, <em>z</em> is the so called backward shift operator which is defined as <em>z</em><sup><em>m</em></sup> <em>X</em><sub><em>t</em></sub> = <em>X</em><sub><em>t</em>-<em>m</em></sub>. The autoregressive operator <em>a</em>(<em>z</em>) of order <em>p</em> is defined as
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img44.gif" alt="\begin{displaymath} a(z) = 1 - a_1 z - a_2 z^2 - \ldots - a_p z^p\end{displaymath}" width="249" height="29" /></p>
the moving average operator <em>b</em>(<em>z</em>) of order <em>q</em> is defined as
<p align="CENTER"><img src="http://rudorfer.homedns.org/apl94/img45.gif" alt="\begin{displaymath} b(z) = 1 - b_1 z - b_2 z^2 - \ldots - b_q z^q\end{displaymath}" width="241" height="29" /></p>
<p>We fitted an ARIMA model for each time series using the SAS system and let it predict the next 20 observations of the time series. The last 20 observations were dropped from the time series and used to calculate the prediction error of the models.</p>
<p>The following ARIMA models were calculated for the airline passenger time series (after a logarithmic transformation):</p>
<p align="CENTER">(1-<em>z</em>)(1-<em>z<sup>12</sup></em>)<em>X</em><sub><em>t</em></sub> = (1 - 0.24169<em>z</em> - 0.47962<em>z<sup>12</sup></em>) <em>U</em><sub><em>t</em></sub></p>
and for the IBM time series:
<p align="CENTER">(1-<em>z</em>) <em>X</em><sub><em>t</em></sub> = (1 - 0.10538<em>z</em>) <em>U</em><sub><em>t</em></sub></p>
<p>As an opponent for the ARIMA modeling technique, we selected those networks that delivered the smallest forecast error <em>s</em><sub><em>f</em></sub> for the respective time series data:</p>
<div align="CENTER"><a name="309"> </a>
<table border="1" cellpadding="3"><caption><strong>Figure:</strong> Number of input and hidden units, IBM share price, forecasting quality</caption>
<tbody>
<tr valign="TOP">
<td align="LEFT" nowrap="nowrap">series</td>
<td align="RIGHT" nowrap="nowrap"><img src="http://rudorfer.homedns.org/apl94/img29.gif" alt="$\eta$" width="12" height="27" align="MIDDLE" border="0" /></td>
<td align="RIGHT" nowrap="nowrap"><img src="http://rudorfer.homedns.org/apl94/img31.gif" alt="$\alpha$" width="14" height="14" align="BOTTOM" border="0" /></td>
<td align="RIGHT" nowrap="nowrap"># input</td>
<td align="RIGHT" nowrap="nowrap"># hidden</td>
</tr>
<tr valign="TOP">
<td align="LEFT" nowrap="nowrap"> </td>
<td align="RIGHT" nowrap="nowrap"> </td>
<td align="RIGHT" nowrap="nowrap"> </td>
<td align="RIGHT" nowrap="nowrap">units</td>
<td align="RIGHT" nowrap="nowrap">units</td>
</tr>
<tr valign="TOP">
<td align="LEFT" nowrap="nowrap">airline</td>
<td align="RIGHT" nowrap="nowrap">0.1</td>
<td align="RIGHT" nowrap="nowrap">0.9</td>
<td align="RIGHT" nowrap="nowrap">70</td>
<td align="RIGHT" nowrap="nowrap">45</td>
</tr>
<tr valign="TOP">
<td align="LEFT" nowrap="nowrap">IBM</td>
<td align="RIGHT" nowrap="nowrap">0.1</td>
<td align="RIGHT" nowrap="nowrap">0.9</td>
<td align="RIGHT" nowrap="nowrap">80</td>
<td align="RIGHT" nowrap="nowrap">30</td>
</tr>
</tbody>
</table>
</div>
<p>In Table <a href="http://rudorfer.homedns.org/apl94/node9.html#tab:vergleich">2</a> the prediction errors for the artificial neural network (ANN), the artificial neural network using the logarithmic and <img src="http://rudorfer.homedns.org/apl94/img19.gif" alt="$\nabla$" width="17" height="15" align="BOTTOM" border="0" /> transformation (ANN log,<img src="http://rudorfer.homedns.org/apl94/img19.gif" alt="$\nabla$" width="17" height="15" align="BOTTOM" border="0" />) and the ARIMA model are compared: The artificial neural network using the logarithmic and <img src="http://rudorfer.homedns.org/apl94/img19.gif" alt="$\nabla$" width="17" height="15" align="BOTTOM" border="0" /> transformed time series outperformed the ARIMA models for both time series, whereas the ``simple'' artificial neural network predicted more accurately only for the IBM shares time series. This behavior can be explained as follows: the larger data range of the airline passenger time series leads to a loss of precision for the untransformed input set. Differencing and logarithmic transformations helped to eliminate the trend and mapped the time series data into a smaller range.</p>
<p> </p>
<a name="tab:vergleich"> </a><a name="270"> </a>
<table><caption><strong>Figure:</strong> Forecasting errors for ANN and ARIMA model</caption>
<tbody>
<tr>
<td>
<p><img src="http://rudorfer.homedns.org/apl94/img46.gif" alt="\begin{figure} \leavevmode \begin{center} \begin{tabular} {\vert l\vert r\ver... ...e price & 7.97 & 7.70 & 11.35 \\ \hline \end{tabular} \end{center}\end{figure}" width="386" height="86" /></p>
<h1><a name="SECTION00070000000000000000"> Dyalog APL ANN Code</a></h1>
<a name="chap:code"> </a> These functions present a complete implementation of the definitions given in the introduction. The main function NET learns the well known XOR (exclusive or) mapping. For reasons of simplicity, the input-output mapping of the XOR function is coded into the function NET.
<p> </p>
<hr /><!--Table of Child-Links--> <a name="CHILD_LINKS"> </a>
<ul>
<li><a href="http://rudorfer.homedns.org/apl94/node11.html#SECTION00071000000000000000" name="tex2html138"> BACKWARD</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node12.html#SECTION00072000000000000000" name="tex2html139"> FORWARD</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node13.html#SECTION00073000000000000000" name="tex2html140"> GRADIENT</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node14.html#SECTION00074000000000000000" name="tex2html141"> NET</a></li>
<li><a href="http://rudorfer.homedns.org/apl94/node15.html#SECTION00075000000000000000" name="tex2html142"> SQUASH</a></li>
</ul>
<h2><a name="SECTION00071000000000000000"> BACKWARD</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img47.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLZ \APLleftarro... ...PLN\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="340" height="326" align="BOTTOM" border="0" />
<h2><a name="SECTION00072000000000000000"> FORWARD</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img48.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLZ \APLleftarro... ...PLN\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="342" height="189" align="BOTTOM" border="0" />
<h2><a name="SECTION00073000000000000000"> GRADIENT</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img49.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLZ \APLleftarro... ...PLX\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="328" height="104" align="BOTTOM" border="0" />
<h2><a name="SECTION00074000000000000000"> NET</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img50.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLN\APLE\APLT \A... ...PLN\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="349" height="461" align="BOTTOM" border="0" />
<h2><a name="SECTION00075000000000000000"> SQUASH</a></h2>
<br /> <img src="http://rudorfer.homedns.org/apl94/img51.gif" alt="\begin{APLfns} \tt \begin{APLfnsline} {}{ \APLdel } \APLspace\APLZ \APLleftarro... ...PLA\end{APLfnsline}\begin{APLfnsline} {}{ \APLdel }\end{APLfnsline}\end{APLfns}" width="172" height="58" align="BOTTOM" border="0" /> <br />
<h1><a name="SECTION00080000000000000000">Conclusion and Further Work</a></h1>
<p>We have presented a forecasting system for univariate time series that uses artificial neural networks. This computing devices proved themselves to be viable alternatives to conventional techniques. The system can be used in conjunction with other techniques for time series analysis or as a stand-alone tool.</p>
<p>Further work will include the comparison with other time series analysis techniques, development of hybrid techniques that combine the strength of conventional approaches with artificial neural networks and the application of our system to multivariate time series.</p>
<h2><a name="SECTIONREF">References</a></h2>
<dl compact="compact"><dt><a name="alf91"><strong>Alf91</strong></a></dt><dd>M. Alfonseca. <br />Advanced applications of APL: logic programming, neural networks and hypertext. <br /><em>IBM Systems Journal</em>, 30(4):543-553, 1991.</dd><dt><a name="box-jenkins"><strong>BJ76</strong></a></dt><dd>George E. P. Box and Gwilym M. Jenkins. <br /><em>Time Series Analysis - forecasting and control</em>. <br />Series in Time Series Analysis. Holden-Day, 500 Sansome Street, San Franciso, California, 1976.</dd><dt><a name="chatfield"><strong>Cha91</strong></a></dt><dd>E. Chatfield. <br /><em>The Analysis of Time Series</em>. <br />Chapman and Hall, New York, fourth edition, 1991.</dd><dt><a name="ch92"><strong>CMMR92</strong></a></dt><dd>Kanad Chakraborty, Kishan Mehrota, Chilukuri K. Mohan, and Sanjay Ranka. <br />Forecasting the Behaviour of Multivariate Time Series Using Neural Networds. <br /><em>Neural Networks</em>, 5:961-970, 1992.</dd><dt><a name="dyalogman"><strong>Dya91</strong></a></dt><dd>Dyadic Systems Limited, Riverside View, Basing Road, Old Basing, Basingstoke, Hampshire RG24 0AL, England. <br /><em>Dyalog Apl Users Guide</em>, 1991.</dd><dt><a name="evans91"><strong>ES91</strong></a></dt><dd>Richard M. Evans and Alvin J. Surkan. <br />Relating Numbers of Processing Elements in a Sparse Distributed Memory Model to Learning Rate and Generalization. <br /><em>ACM APL Quote Quad</em>, 21(4):166-173, 1991.</dd><dt><a name="hertzkroghpalmer"><strong>HKP91</strong></a></dt><dd>John Hertz, Anders Krogh, and Richard G. Palmer. <br /><em>Introduction to the Theory od Neural Computation</em>. <br />Addison Wesley, Redwood City, California, 1991.</dd><dt><a name="hornik"><strong>HSW89</strong></a></dt><dd>Kurt Hornik, Maxwell Stinchcombe, and Halbert White. <br />Multilayer Feedforward Networks are Universal Approximators. <br /><em>Neural Networks</em>, 2:359-366, 1989.</dd><dt><a name="peele81"><strong>Pee81</strong></a></dt><dd>Howard A. Peele. <br />Teaching A Topic in Cybernetics with APL: An Introduction to Neural Net Modelling. <br /><em>ACM APL Quote Quad</em>, 12(1):235-239, 1981.</dd><dt><a name="rummel"><strong>RHW86</strong></a></dt><dd>David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. <br />Learning representations by back-propagating errors. <br /><em>Nature</em>, 323(9):533-536, October 1986.</dd><dt><a name="turingnn"><strong>SS91</strong></a></dt><dd>Hava Siegelmann and Eduardo D. Sontag. <br />Neural Nets Are Universal Computing Devices. <br />Technical Report SYSCON-91-08, Rutgers Center for Systems and Control, May 1991.</dd><dt><a name="sku93"><strong>SS93</strong></a></dt><dd>Alexei N. Skurihin and Alvin J. Surkan. <br />Identification of Parallelism in Neural Networks by Simulation with Language J. <br /><em>ACM APL Quote Quad</em>, 24(1):230-237, 1993.</dd><dt><a name="white88"><strong>Whi88</strong></a></dt><dd>Halbert White. <br />Economic prediction using neural networks: the case of ibm daily stock returns. <br />In <em>Proceedings of the IEEE International Conference on Neural Networks</em>, pages II-451-II-459, 1988.</dd></dl><hr /><address> </address>
<table>
<tbody>
<tr>
<td> </td>
<td><em>© 2010 <a href="http://www.ai.wu.ac.at/mitarbeiter/rudorfer.html">Gottfried Rudorfer</a>, © 1994 ACM APL Quote Quad, 1515 Broadway, New York, N.Y. 10036, <a href="http://wwwai.wu-wien.ac.at/">Abteilung für Angewandte Informatik</a>, <a href="http://www.wu-wien.ac.at/">Wirtschaftsuniversität Wien</a>, 3/23/1998</em></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<!--End of Table of Child-Links-->