# Neural networks: Nodes and hidden layers

To build a [**neural network**](https://developers.google.com/machine-learning/glossary#neural_network)
that learns [**nonlinearities**](https://developers.google.com/machine-learning/glossary#nonlinear),
begin with the following familiar model structure: a
[**linear model**](https://developers.google.com/machine-learning/glossary#linear-model) of the form
$y' = b + w_1x_1 + w_2x_2 + w_3x_3$.

We can visualize this equation as shown below, where $x_1$,
$x_2$, and $x_3$ are our three input nodes (in blue), and $y'$ is our output node
(in green).

## Exercise 1

In the model above, the [**weight**](https://developers.google.com/machine-learning/glossary#weight) and
[**bias**](https://developers.google.com/machine-learning/glossary#bias) values have been randomly
initialized. Perform the following tasks to familiarize yourself with the
interface and explore the linear model. You can
ignore the *Activation Function* dropdown for now; we'll discuss this
topic later on in the module.

1. Click the Play (▶️) button above the network to calculate the value of
   the output node for the input values $x_1 = 1.00$, $x_2 = 2.00$, and
   $x_3 = 3.00$.

2. Click the second node in the
   [**input layer**](https://developers.google.com/machine-learning/glossary#input-layer), and increase
   the value from 2.00
   to 2.50. Note that the value of the output node changes. Select the output
   nodes (in green) and review the *Calculations* panel to see how the output
   value was calculated.

   > [!NOTE]
   > **Notes about calculations:**
   > - Values displayed are rounded to the hundredths place.
   > - The `Linear()` function simply returns the value it is passed.

3. Click the output node (in green) to see the weight ($w_1$, $w_2$, $w_3$) and
   bias ($b$) parameter values. Decrease the weight value for
   $w_3$ (again, note that the value of the output node and the calculations below
   have changed). Then, increase the bias value. Review how these changes
   have affected the model output.

## Adding layers to the network

Note that when you adjusted the weight and bias values of the network in
[Exercise 1](https://developers.google.com/machine-learning/crash-course/neural-networks/nodes-hidden-layers#exercise_1), that didn't change the overall mathematical
relationship between input and output. Our model is still a linear model.

But what if we add another layer to the network, in between the input layer
and the output layer? In neural network terminology, additional layers between
the input layer and the output layer are called
[**hidden layers**](https://developers.google.com/machine-learning/glossary#hidden_layer), and the nodes
in these layers are called
[**neurons**](https://developers.google.com/machine-learning/glossary#neuron).

The value of each neuron in the hidden layer is calculated the same way as the
output of a linear model: take the sum of the product of each of its inputs
(the neurons in the previous network layer) and a unique weight parameter,
plus the bias. Similarly, the neurons in the next layer (here, the output layer)
are calculated using the hidden layer's neuron values as inputs.

This new hidden layer allows our model to recombine the input data using another
set of parameters. Can this help our model learn nonlinear relationships?

## Exercise 2

We've added a hidden layer containing four neurons to the model.

Click the Play (▶️) button above the network to calculate the value of
the four hidden-layer nodes and the output node for the input values
$x_1 = 1.00$, $x_2 = 2.00$, and $x_3 = 3.00$.

Then explore the model, and use it to answer the following questions.
How many [**parameters**](https://developers.google.com/machine-learning/glossary#parameter) (weights and biases) does this neural network model have? 4 Our original model in [Exercise 1](https://developers.google.com/machine-learning/crash-course/neural-networks/nodes-hidden-layers#exercise-1) had four parameters: w~11~, w~21~, w~31~, and b. Because this model contains a hidden layer, there are more parameters. 12 Note that the total number of parameters includes both the parameters used to calculate the node values in the hidden layer from the input values, and the parameters used to calculate the output value from the node values in the hidden layer. 16 Note that the total number of parameters includes both the weight parameters and the bias parameters. 21 There are 4 parameters used to calculate each of the 4 node values in the hidden layer---3 weights (one for each input value) and a bias---which sums to 16 parameters. Then there are 5 parameters used to calculate the output value: 4 weights (one for each node in the hidden layer) and a bias. In total, this neural network has 21 parameters. Try modifying the model parameters, and observe the effect on the
hidden-layer node values and the output value (you can review the
Calculations panel below to see how these values were
calculated).

Can this model learn nonlinearities?
Yes Click on each of the nodes in the hidden layer and the output node, and review the calculations below. What do you notice about all these calculations? No If you click on each of the nodes in the hidden layer and review
the calculations below, you'll see that all of them are linear
(comprising multiplication and addition operations).

If you then click on the output node and review the calculation
below, you'll see that this calculation is also linear. Linear
calculations performed on the output of linear calculations are
also linear, which means this model cannot learn nonlinearities.

> [!IMPORTANT]
> **Key terms:**
>
> - [Bias](https://developers.google.com/machine-learning/glossary#bias)
> - [Hidden layer](https://developers.google.com/machine-learning/glossary#hidden_layer)
> - [Input layer](https://developers.google.com/machine-learning/glossary#input-layer)
> - [Linear model](https://developers.google.com/machine-learning/glossary#linear-model)
> - [Model](https://developers.google.com/machine-learning/glossary#model)
> - [Neural network](https://developers.google.com/machine-learning/glossary#neural_network)
> - [Neuron](https://developers.google.com/machine-learning/glossary#neuron)
> - [Nonlinear](https://developers.google.com/machine-learning/glossary#nonlinear)
> - [Parameter](https://developers.google.com/machine-learning/glossary#parameter)
> - [Weight](https://developers.google.com/machine-learning/glossary#weight)

[Help Center](https://support.google.com/machinelearningeducation)