Neural Network Architectures

Keywords: neural network; multilayer perceptron; backpropagation; convolutional network; LSTM; self-attention; Transformer

1. The Artificial Neuron

The common building block is a single artificial neuron: it forms a weighted sum of its inputs, adds a bias, and passes the result through a non-linear activation function \( \phi \). For an input vector \( \mathbf{x} \in \mathbb{R}^{d} \), weights \( \mathbf{w} \), and bias \( b \),

The activation \( \phi \) is what makes a stack of neurons more expressive than a single linear map; without it, any depth of network collapses to one linear layer. Common choices are the logistic sigmoid \( \sigma(z) = 1/(1 + e^{-z}) \), the hyperbolic tangent \( \tanh(z) \), and the rectified linear unit \( \operatorname{ReLU}(z) = \max(0, z) \), the last being the default in deep networks because its gradient does not vanish for positive inputs [6].

2. The Fully Connected Network (ANN / MLP)

Stacking neurons into layers, where every neuron in a layer receives all outputs of the previous layer, gives the multilayer perceptron. Writing the weights of layer \( \ell \) as a matrix \( W^{[\ell]} \) and the biases as a vector \( \mathbf{b}^{[\ell]} \), the forward pass is a chain of affine maps and activations:

with \( \hat{\mathbf{y}} = \mathbf{a}^{[L]} \) the output. A network with even one sufficiently wide hidden layer can approximate any continuous function on a bounded domain to arbitrary accuracy, the universal approximation theorem of Cybenko [1] and Hornik [2]; depth makes that approximation reachable with far fewer units. This expressiveness is also the caution: an MLP fits well inside the range of its training data and offers little guarantee outside it, which is exactly why, in the flow stress work, a network is paired with a tensor decomposition that keeps the physical structure visible rather than being used as a stand-alone black box.

3. Training: The Backpropagation Algorithm

Training chooses the weights and biases to minimize a loss \( \mathcal{L}(\hat{\mathbf{y}}, \mathbf{y}) \), for example the mean squared error for regression or the cross-entropy for classification, averaged over the training set. Minimization is by gradient descent, and the gradients are computed efficiently by backpropagation [3], which is the chain rule applied layer by layer from the output backwards. Define the error signal of layer \( \ell \) as \( \boldsymbol\delta^{[\ell]} = \partial \mathcal{L} / \partial \mathbf{z}^{[\ell]} \). At the output layer,

and it propagates backwards through the transpose of each weight matrix:

where \( \odot \) is the elementwise product. The parameter gradients then follow directly,

and each parameter is updated by a step against its gradient, \( W \leftarrow W - \eta\,\partial\mathcal{L}/\partial W \), with learning rate \( \eta \), in practice using a mini-batch stochastic optimizer such as Adam [7]. Equations (3) to (5) are the whole of supervised training; the architectures that follow change only the form of the forward pass, and their gradients are obtained by the same backward recursion, today performed automatically by differentiation frameworks.

4. The Convolutional Neural Network (CNN)

When the input has grid structure, an image, a spectrogram, or a one-dimensional waveform, a fully connected layer is wasteful and ignores the fact that a useful feature (an edge, a burst) looks the same wherever it occurs. A convolutional layer instead slides a small shared filter (kernel) \( K \) across the input, computing a local weighted sum at every position. For a 2D input \( X \) and kernel of size \( (2m{+}1)\times(2n{+}1) \),

Two properties fall out of Eq. (6): weight sharing, because the same kernel is used at every position, so the layer has far fewer parameters than a dense one and detects a feature regardless of where it sits (translation equivariance); and locality, because each output depends only on a small receptive field. A layer applies many kernels to produce a stack of feature maps. Between convolutional layers a pooling operation downsamples each map, for example max-pooling over a \( 2\times2 \) window, \( P_{ij} = \max_{(u,v)\in\text{window}} A_{i+u,\,j+v} \), which shrinks the representation and adds a small tolerance to translation. Stacked convolution-plus-pooling blocks build features from local edges up to whole objects; the design traces to LeCun's LeNet [4] and became dominant after AlexNet [5], with residual connections [8] later enabling very deep networks.

5. The Long Short-Term Memory Network (LSTM)

For sequential data, where order carries meaning, a recurrent network processes one element at a time while carrying a hidden state \( \mathbf{h}_t \) forward. A plain recurrent network struggles to learn long-range dependencies because repeated multiplication through the recurrence makes gradients vanish or explode. The long short-term memory unit of Hochreiter and Schmidhuber [9] solves this by adding a protected cell state \( \mathbf{c}_t \) that information can flow along nearly unchanged, regulated by three multiplicative gates. At each step, given input \( \mathbf{x}_t \) and previous state \( \mathbf{h}_{t-1} \):

The forget gate \( \mathbf{f}_t \) decides what to erase from the cell, the input gate \( \mathbf{i}_t \) what new information to write, and the output gate \( \mathbf{o}_t \) what to expose as the hidden state. Because the cell update in Eq. (7) is additive rather than a repeated matrix product, the gradient can travel many steps without decaying, which is what lets an LSTM relate events far apart in a sequence. A bidirectional LSTM runs two such chains, one forward and one backward, and concatenates their states, so each position sees both past and future context; this is the recurrent half of the CNN-BiLSTM used in the battery acoustic-emission work. A lighter-weight variant, the gated recurrent unit (GRU) [10], merges the gates into two.

6. The Transformer

Recurrence forces sequential processing: step \( t \) cannot be computed until step \( t{-}1 \) is done. The Transformer of Vaswani et al. [11] removes recurrence entirely and lets every position of a sequence interact with every other position directly, through self-attention, which is fully parallel. Each input element is first projected into three vectors, a query \( \mathbf{q} \), a key \( \mathbf{k} \), and a value \( \mathbf{v} \); stacking these over all positions gives matrices \( Q, K, V \). Scaled dot-product attention computes, for every query, a weighted average of all values, weighted by how well the query matches each key:

The product \( QK^{\mathsf T} \) scores every query-key pair; dividing by \( \sqrt{d_k} \), the key dimension, keeps the scores from growing large enough to saturate the softmax; the softmax turns each row into attention weights that sum to one; and multiplying by \( V \) mixes the values accordingly. Running several such attention functions in parallel with different learned projections, then concatenating them, is multi-head attention, which lets the model attend to several kinds of relationship at once:

Because attention alone is order-blind (permuting the inputs permutes the outputs identically), a positional encoding is added to the inputs to inject sequence order. Each Transformer block wraps multi-head attention and a position-wise feed-forward MLP, each with a residual connection [8] and layer normalization. This architecture is the basis of modern large language models and has spread to vision and time-series problems; its cost is quadratic in sequence length, since \( QK^{\mathsf T} \) compares all pairs.

7. Choosing an Architecture

The four architectures embody increasingly weak assumptions about the input, traded for increasingly large data appetite. An MLP suits fixed-length feature vectors with no known structure. A CNN suits data with local, translation-invariant patterns: images, and one-dimensional signals such as stress waves or acoustic-emission bursts. An LSTM suits sequences whose order matters and whose useful dependencies may be short or long. A Transformer suits sequences where long-range, all-to-all relationships dominate and enough data and compute are available to learn them. The choice is a matter of matching the structural prior to the data, not of picking the newest model; a small, well-structured engineering dataset is often served better by a modest CNN or LSTM, or by a physics-structured hybrid, than by a large attention model.

8. Use on This Site

References

1. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4), 303-314.
2. Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2(5), 359-366.
3. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536.
4. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
5. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 1097-1105.
6. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
7. Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR).
8. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770-778).
9. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780.
10. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of EMNLP (pp. 1724-1734).
11. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998-6008.

Back to Technical Notes