Neural Nets

Part I: Square arrays mirrored/copied horizontally or vertically

I.a. In rmirror.m you'll find a simple Octave script that generates a square array of random values, which are mirrored either top-to-bottom or side-to-side.

It takes two values:

X = rmirror(size,direction);

where the resulting array will be 2*size by 2*size, and will be mirrored horizontally if direction is 0, or vertically if direction is 1. Thus

imagesc(rmirror(4,0))	imagesc(rmirror(4,1))

Your task is to build a classifier to determine whether the array is mirrored horizontally or vertically.

This is a trivial coding problem in a language like Octave or Matlab, but now we're going to ask you to build a pseudo-neural network that solves the problem. The input will be the value of the matrix elements, and the binary determination of the direction of symmetry will determined by whether an output value is positive or negative.

To make it easier, start with the simplest possible 2x2 array, i.e. with examples generated by rmirror(1,0) or rmirror(1,1). Thus there will be four inputs and one output -- in between the input and the output, you can put any combination of weighted links and point non-linearities that you like.

You should configure your "network" as a set of vectors, some of which represent the value of (different layers of) nodes, while others represent the "weights" connecting one layer to another. A suggestion would be to use RELU units ("rectified linear units") as the non-linearities -- this is the trivial function f(x) = max(0,x).

Your program  should specify (and implement) the number and connectivity of the layers, and the weights connecting them. No other calculations should be performed.

Verify that your program correctly classifies 2x2 matrices.

I.b.  You can add noise to these matrices via e.g. rmirror(1,0) + alpha*unifrnd(0,1,[2 2]), where the amount of noise might be relatively small (relatively small values of alpha) or large (large values alpha). How much noise can you add before the classification of the original matrix symmetry is no longer reliable?

I.c.  Now create an analogous program that learns the right weights to solve the trivial 2x2 version of the problem. You can do this is one of three ways:

1. Write your own code in Octave or Matlab to implement back-propagation weight training;
2. Use the (very simple) Octave nnet package;
3. Use the (more elaborate and capable) Matlab Neural Network Toolbox. (Tutorials here.)

In all cases, you'll generate arbitrary amount of training data using an appropriate modification of the rmirror() code.

II.a. Modify your solution to I.c. to learn weights for the 4x4 problem.

Extra credit: What happens to the noise resistance in this case?

II.b.  Try the resulting net out on 4x4 matrices generated by cmirror.m, which replicates the random values top-to-bottom or side-to-side by copying (in order) rather than mirroring (in reverse order). Does it work? If not, why not?

II.c. Retrain your network using a mixture of rmirror() and cmirror() random inputs. Does this work? Has the network learned to generalize, or has it learned to solve two independent problems? How can you tell?

Extra credit: Modify rmirror() to use a randomly selected offset and scaling factor in creating the redundant (top-to-bottom or side-to-side) values. Thus the code for horizontal reflection might become

  x(1:size2,size2:-1:(size+1)) = OFFSET + SCALE*x(1:size2,1:size);

where OFFSET and SCALE are selected randomly for each matrix creation. Will the network that you trained in II.a. still work? If not, can you create one that will? Does this merely require retraining or do you need to modify the architecture? Why or why not?

III.a. (Optional)  Consider a time-series consisting of a periodic signal whose period either monotonically increase or decreases. One particularly simple version of such signals can be generated via the simple function ramp.m. (You will also need the function afwosc.m) Thus

soundsc(ramp(100,200,100));

will generate this sound.

How might you program a network to classify such sequences?

For a fixed-length sequence, Stephen Isard points out that you could estimate pitch change by using the network to calculate a cepstrum (power spectrum of the log of the power spectrum) for the starting and the ending parts of the sequence. In principle a trainable network might learn that approach -- among many others.

How could you design a trainable network that could learn to distinguish rising from falling sequences -- of arbitrary length and arbitrary frequency ranges? For example, you might hope that a recurrent network would learn to do serial cross-correlation. What if you added variation in amplitude contour and timbre?