next up previous contents
Next: Training the System Up: A New Modular Neural Previous: The Evolution of the

The Network Architecture

The proposed network system consists of a layer of input modules and an additional decision module. All sub-networks are MLPs. Only the number of inputs and the number of outputs of the module is determined by the system. The internal structure, such as the number of hidden layers and the number of neurons in each hidden layer can be chosen independent of the overall architecture.

Each input variable is connected to only one of the input modules. These connections are chosen at random. The outputs of all input modules are connected to the decision network. In the discussion that follows the dimension of the input vector is denoted by l and the number of classes by k.

Figure 5.2:  The Proposed Modular Neural Network Architecture.

To determine a modular network it is necessary to specify either the number of inputs per input module or the number of input modules. These parameters are dependent on each other. It is assumed here that the number of inputs per module in the first layer is chosen as n; the number of input modules in the input layer can therefore be calculated as m = {ln} .

  It is further assumed that l = m n. If this is not the case the spare inputs can be connected to constant inputs; in the implementation of the model all `free' inputs were connected to the constant value `0'. Alternatively, it would be possible to alter the size of one module or of a group of modules.

Each module in the input layer can have either k or log2k outputs. The network with k intermediate outputs is refered to as large intermediate representation. It only is useful if the number of classes is very small. For problems with a larger number of classes the small intermediate representation (log2k ) is more appropriate.

From an information theory point of view the small intermediate representation should be sufficient because only this number of output neurons is required to represent all the classes in a binary code.

The decision network has (m k) or (m log2k ) inputs, dependent on the intermediate representation used. The number of outputs is k, one output neuron for each class. The structure of the modular network is depicted in Figure 5.2 using a small intermediate representation. The function π: X X gives a permutation on X = {1 &ldots;l }. This permutation is randomly chosen and constant for a network.


Definition: A Module
A module is a multilayer feedforward neural network defined by a 3-tuple:

M=(a, b, H)

Where a is the number of inputs of the module, b is the number of output nodes, and H is a list containing the numbers of neurons in each of the hidden layers.

A multilayer Perceptron module M with eight inputs, twelve neurons in the first hidden layers, ten neurons in the second hidden layer, and four outputs is described as: M = (8, 4, [12,10]).

Definition: A Modular Neural Network
A modular neural network is a set of interconnected modules defined by a 7-tuple.

N=(l, k, m, r, π, I, D)

Where l is the number of inputs, k the number of classes, m the number of modules in the input layer, r is the type of the intermediate representation (r ∈{small, large }), π is the permutation function, I is the input layer module, and D is the decision module.

A modular network with 210 inputs, four classes, 15 input modules (each having 14 inputs, one hidden layer with six neurons and two outputs), a small intermediate representation, a permutation function p, and a decision network (with 30 inputs, one hidden layer with ten neurons and 4 output neurons) is described as: N=(210, 4, 15, small, p, (14,2,[6]), (30,4,[10])).

next up previous contents
Next: Training the System Up: A New Modular Neural Previous: The Evolution of the

Albrecht Schmidt
Mit Okt 4 16:45:34 CEST 2000