Splitting-up the training set into subsets can also bring problems. The number of equal input vectors with different possible output values may increase, especially for modules with a small number of input variables. Consider the worst case: a 4-Bit-Parity problem:
Original Set | Set MLP | Set MLP | |||
0 0 0 0 | 0 | 0 0 | 0 | 0 0 | 0 |
0 0 0 1 | 1 | 0 0 | 1 | 0 1 | 1 |
0 0 1 0 | 1 | 0 0 | 1 | 1 0 | 1 |
0 0 1 1 | 0 | 0 0 | 0 | 1 1 | 0 |
0 1 0 0 | 1 | 0 1 | 1 | 0 0 | 1 |
0 1 0 1 | 0 | 0 1 | 0 | 0 1 | 0 |
0 1 1 0 | 0 | 0 1 | 0 | 1 0 | 0 |
0 1 1 1 | 1 | 0 1 | 1 | 1 1 | 1 |
1 0 0 0 | 1 | 1 0 | 1 | 0 0 | 1 |
1 0 0 1 | 0 | 1 0 | 0 | 0 1 | 0 |
1 0 1 0 | 0 | 1 0 | 0 | 1 0 | 0 |
1 0 1 1 | 1 | 1 0 | 1 | 1 1 | 1 |
1 1 0 0 | 0 | 1 1 | 0 | 0 0 | 0 |
1 1 0 1 | 1 | 1 1 | 1 | 0 1 | 1 |
1 1 1 0 | 1 | 1 1 | 1 | 1 0 | 1 |
1 1 1 1 | 0 | 1 1 | 0 | 1 1 | 0 |
To discuss this problem more general the following definition is necessary. is the probability of the output variable having the value . And is the conditional probability of if .
Definition: A Set of Statistically Neutral Variables
Consider the function .
A subset of the input with is called
statistically neutral if the knowledge of the values of these variables
does not increase the knowledge of the result .
Formally: The set of input variables is statistically neutral if:
If the whole set of input variables and all possible subsets for all modules are statistically neutral, the network can not learn the task. If only some of the modules are supplied with a statistically neutral set of input variables the network may perform satisfactory.
In [ston95] it is shown that monolithic multilayer feedforward networks can learn statistically neutral problems but they can not generalize on them.
This situation sounds very unlikely in real world data, and the expectation was observed during experimentation.
Especially in tasks with a large input space, such as picture recognition, this problem can be ignored. In tasks with a small number of input attributes, one way to reduce the possibility of having statistically neutral input sets is to recode the data in a sparse code.