Charta is on Google Play: practise on the go. Get the app
Charta

Describe how neural networks are put together and how the values of their weights are arrived at.

FRM® Part I · Quantitative Analysis · Machine Learning and Prediction · FRM1.QA.15.h

The idea

A feed-forward neural network is arranged in layers: an input layer, one or more hidden layers and an output layer. Each neuron computes a weighted sum of the previous layer's outputs plus a bias. z = w₁ × x₁ + w₂ × x₂ + … + b. The neuron then passes z through an activation function such as the sigmoid, tanh or ReLU. The activation supplies non-linearity. Without it, stacked layers collapse into a single linear model. The weights are not solved in closed form. They start at small random values, and a forward pass produces predictions and a loss such as squared error or cross-entropy. Backpropagation uses the chain rule to get the loss gradient for each weight. Gradient descent then updates each weight. New w = w − learning rate × gradient. This repeats over many epochs. Parameters = inputs × neurons + neurons. This counts the weights and biases in a layer. Overfitting is controlled by early stopping, dropout or regularization.

Formulas

Parameters in a fully connected layer

parameters = inputs × neurons + neurons

inputs
the number of values feeding the layer
neurons
the number of neurons in the layer, each with one bias

Add the layers together for the whole network, remembering a bias for every hidden and output neuron.

Gradient descent weight update

new w = w − learning rate × gradient

w
the current weight
learning rate
the step size
gradient
the derivative of the loss with respect to w, from backpropagation

Subtract the gradient: a negative gradient raises the weight. Repeated over many epochs, not solved in closed form.

A practice question

From the question bank, with the reason each wrong option is wrong.

In a feed-forward neural network, a hidden neuron computes:

  1. a weighted sum of its inputs plus a bias, which is then passed through a non-linear activation function. - correct
  2. a weighted sum of its inputs plus a bias with no activation, so that deeper networks capture non-linear patterns.Why not: Without an activation, stacked layers collapse into one linear model and add no non-linearity.
  3. the majority class among the training points closest to its inputs.Why not: Describes a nearest-neighbour vote, not a neuron.
  4. the plain average of its inputs, with weights applied only at the output layer.Why not: Every neuron has its own weights; the weights are not confined to the output layer.

A neuron computes z = w . x + b and outputs g(z). The non-linear g gives a network its power.

A harder one

A single output neuron has inputs x1 = 0.5 and x2 = 2.0, weights 0.4 and -0.3, a bias of 0.1 and a sigmoid activation. The neuron's output is closest to:

  1. 0.40Why not: Omits the bias, using z = -0.4, instead of z = -0.3.
  2. 0.57Why not: Gets the sign of z wrong, using +0.3 rather than -0.3.
  3. 0.71Why not: Ignores the signs of the weights, using z = 0.2 + 0.6 + 0.1 = 0.9.
  4. 0.43 - correct

z = 0.4 x 0.5 - 0.3 x 2.0 + 0.1 = -0.3. Sigmoid(-0.3) = 1/(1 + e^0.3) = 0.426, about 0.43.

The hardest tier

A feed-forward network has 10 inputs, two hidden layers with 8 and 4 neurons, and one output neuron. Layers are fully connected and every neuron in the hidden and output layers has a bias. The number of trainable parameters is:

  1. 116Why not: Counts weights only (80 + 32 + 4) and leaves out all of the biases.
  2. 129 - correct
  3. 128Why not: Includes the hidden-layer biases but leaves out the bias of the output neuron.
  4. 139Why not: Adds a bias for each of the 10 input nodes, which do not compute anything.

Layer 1: 10 x 8 + 8 = 88. Layer 2: 8 x 4 + 4 = 36. Output: 4 x 1 + 1 = 5. Total = 129.

In the app: 6 practice questions on this outcome, spaced review that brings it back before you forget it.

Start free →

More in Machine Learning and Prediction