[TUT] Neural Network Basics

"Basics" because that's all I know right now!


The Very Basics Part 1- A Neuron

At the most basic level, all a neuron does is take inputs, multiply each input by a certain number (may be different number for each input), add up these numbers, and pass them through an equation.

Let's say we have three inputs: x1, x2, x3
Each input has a corresponding "weight", by which each input is multiplied. When a Neuron "learns", the weights are the only things to change. Our weights will be w1, w2, and w3.

So out little neuron would take the sum of each product:
(x1*w1) + (x2*w2) + (x3*w3)
We'll call this sum, s

And the neuron passes this sum, s, through a function, which we'll call f(s). The neuron sends f(s) on its merry little way.
Each neuron returns one value, but may receive multiple inputs, as long as the number of weights within that neuron is equal to the number of inputs to be received.


The Very Basics Part 2- A Neuron Layer

A Neuron Layer is nothing more than a group of neurons. As a layer, each neuron within the group receives the SAME SET of inputs. Thus each neuron within a layer has the SAME NUMBER of weights (each individual weight may hold different values).

If a Neuron1 receives the inputs:
1, 4.6, 0.71
Then Neuron2 within that same layer will receive the inputs:
1, 4.6, 0.71
However Neuron1 may have weights:
0.91, 2.049, 6.012
And Neuron2 can have weights:
4.137, 0.012, 1.123



The Very Basics Part 3- A Neural Network

A Neural Network consists of several Neuron Layers. Every Neural Network has an Input Layer and an Output Layer. Most Networks also have one or several Hidden Layers.
An Input Layer is used to take in inputs. The input(s) in this layer have a weight of 1 and do not pass the weighted sum (sum of the products of inputs * weights) through any equation. This means that the inputs received by this layer are not altered in any way, and they are sent to the next layers just as they are received. The Input Layer has as many Neurons as inputs tht the Network should receive.
An Output Layer, unlike the Input Layer, acts as a regular Layer- it finds the weighted sum of the inputs and passes the results on. The is the last layer of a simple Neural Network. In this layer, there are as many neurons as are outputs of the Network.
A Hidden Layer, like the output layer, passes the weighted sum through an equation and returns the output. each neuron generates an output. The set of these outputs are then released to the next layer as the inputs of that layer.
"Teaching" a Neural Network means adjusting the weights so that the output of the networks approaches the desired output.


If anyone wants to know some algorithms to "teach" the network, I will post what I know ^^ (only backpropagation and evolution right now, I'm still learning).

Code:

#include <vector>
#include <cstdlib>
#include <math.h>

using namespace std;





#define NUM_INPUTS 1
#define NUM_OUTPUTS 1
#define NUM_HIDDEN_LAYERS 1
#define NEURONS_PER_HIDDEN_LAYER 6


inline double RandFloat() //returns a random number from 0 to 1
{
	return (rand() / ((double)RAND_MAX));
};

inline double RandClamp() //returns a random number from -1 to 1
{
	return (RandFloat() - RandFloat());
};

inline double Sigmoid(double x) //the weighted sum goes through this equation
{
	return (1./(1.+exp(-x)));
};


struct Weight //standard weight
{
	double *input; //memory address of the input, will be the output of the corresponding neuron of the previous layer
	double weight; //manipulates the output of the neuron. This is multilies by *input.
	
	Weight(){weight = RandClamp();};
	double output(){return ((*input)*weight);}; //self explanitory
};



struct Neuron
{
	int num_weights; //number of weights in this neuron, equal to the number of NEURONS in the previous layer
	vector<Weight> vec_weights; //vector of weights, has num_weights number of weights
	double output; //holds the output of this neuron- add up all the (weights * input) and pass that (weighted sum) through the
					//equation
	Neuron();
	double Output(); //returns the output of this neuron, also sets the value for the variable output
};

Neuron::Neuron() //initializing a neuron
{	
	output = 0.;
};


double Neuron::Output()
{
	double total_sum=0.;
	for(int i=1; i<=num_weights; i++)			//adds up all the individual (wieght * input)s and
		total_sum += vec_weights[i-1].output();
	output = Sigmoid(total_sum);				//sends it through the equation
	return output;
};


struct Neuron_Layer
{
	int num_neurons; //number of neurons in this layer
	vector<Neuron> vec_neurons; //vector of all neurons
	vector<double> output; //stores the output of each neuron in this layer

	Neuron_Layer(int n_neurons);
	void Output(); //finds the output for each layer and sets the values for vec_output
};

Neuron_Layer::Neuron_Layer(int n_neurons) //initialize Neuron_Layer
{
	num_neurons = n_neurons;
	for(int i=1; i<=num_neurons; i++) //for every integer in num_neuron
	{
		vec_neurons.push_back(Neuron()); //create a new Neuron for this layer
		output.push_back(0.);				//andpush back the vector of output back by 1
	};
};


void Neuron_Layer::Output()
{
	for(int i=1; i<=num_neurons; i++)			//for every neuron in this layer
		output[i-1]=vec_neurons[i-1].Output();	//find the output of that neuron and put that value
};													//into the vector of outputs for this layer


struct Neural_Net
{
	Neural_Net();
	vector<Neuron_Layer> vec_layers; //create a vector of Neuron_Layers
	vector<double> input; //set of inputs for the network
	vector<double> output; //set of outputs generated by the network

	vector<double> f(vector<double> x); //generates the ouput(s) for this network
//	void Update(vector<double> desired_outputs); //a fuction to update each weight in the network 
};

Neural_Net::Neural_Net() //iitialize the Neural_Net
{
	for(int i=1; i<=NUM_OUTPUTS; i++) //for every output of the NETWORK
		output.push_back(0.); //push back the vector of outputs
	for(i=1; i<=NUM_INPUTS; i++) //for every input of this network
		input.push_back(0.); //push back the vector of inputs

	vec_layers.push_back(Neuron_Layer(NUM_INPUTS)); //create a neuron layer, it will have NUM_INPUTS number of neurons
	for(i=1; i<= NUM_HIDDEN_LAYERS+1; i++) //for every desired (hidden layer+1, the +1 is for the output layer
	{
		if(i==NUM_HIDDEN_LAYERS+1) //if this layer sould be the output layer
			vec_layers.push_back(Neuron_Layer(NUM_OUTPUTS)); //create a layer with NUM_OUTPUTS number of neurons
		else //otherwise if this is a hidden layer, not a output layer
			vec_layers.push_back(Neuron_Layer(NEURONS_PER_HIDDEN_LAYER)); //create a layer with 
	};														//NEURONS_PER_HIDDEN_LAYER (predefined number) number of neurons

	for(i=1; i<=NUM_HIDDEN_LAYERS+1; i++) //for every hidden_layer +1 (hidden layer and output layer)
		for(int j=0; j<= vec_layers[i].num_neurons-1; j++) //for every neuron within that layer
		{
			vec_layers[i].vec_neurons[j].num_weights = vec_layers[i-1].num_neurons; //set num_weights for each neuron in that layer to be equal to the number of neurons in the previous layer
			for(int k=0; k<=vec_layers[i].vec_neurons[j].num_weights-1; k++) //and for every weight in that neuron
			{
				vec_layers[i].vec_neurons[j].vec_weights.push_back(Weight()); //add a weight to the vector of weights (which previously held none)
				vec_layers[i].vec_neurons[j].vec_weights[k].input = &(vec_layers[i-1].vec_neurons[k].output); //and set the location
			};															//of its input value to the location the [corresponding neuron of the previous layer]
		};																			//bracketed for readability

};


vector<double> Neural_Net::f(vector<double> x) //generated the output of the network
{
	input = x; //the vector holding the input values fr the network is now defined
	for(int i=1; i<=NUM_INPUTS; i++) //for every input
		vec_layers[0].vec_neurons[i-1].output = x[i-1]; //set the outputs of the [input layer neurons] the the correct x-value (which is given)
	for(i=1; i<=NUM_HIDDEN_LAYERS+1; i++) //then for every hidden layer+1 (hidden layers and the output layer)
		vec_layers[i].Output(); //generate the output for that layer
	output = vec_layers[NUM_HIDDEN_LAYERS+1].output; //set the values for the vec_output in this network as the outputs of the output (last) layer
	return output; //return the outputs of the network
};

 

<3 neural networks. I had an excellent book on this called "The Age of Spiritual Machines." Wish I still had it. Thanks for the tutorial I'm going to have to finish learning C/C++ now.

 

i understood the first 2 things

 

Then part art you didn't understand is Part 3?

A Neural Network is a group of Neuron Layers. The Network must have 2 Layers at least.
-The Input Layer, the first layer, will store the inputs, which wil be given to the network.
-The Outpt Layer, the last layer, will hold the outputs of the Network.
Any additional layers to theNetwork are called a Hidden Layers, which perform the same function as Output Layers. A Hidden Layer's (as well as an Output Layer's) inputs are made up up the PREVIOUS layer's output.

Take a look at the values the structure Neural_Net holds:

Code:

	vector<Neuron_Layer> vec_layers;
	vector<double> input;
	vector<double> output;

vec_layers s a vector of all the Neuron Layers of the Neural Network.
"vector<double> input" stores the values going into the network. These values are given to the input layers, the first Neuron_Layer in vec_layers
"vector<double> output" holds the output values of the neurons in the Output Layer, the last layer of vec_layers.

look at this picture:


Don't try to follow each arrow, but see how many arrows go out of each neuron in the input layer. Now see how many arrows end up in EACH neuron of the hidden layer.
See how many leave each neuron of the hidden layer. See how many end up in each neuron of the output layer.
This example receives 5 inputs, and give out 2 outputs. It has 1 hidden layer. Remember, a network can have as many hidden layers as you want to give it, including 0, but it must have an input layer and an output layer.

In the above picture, lets say each arrow represents a number.
Each neuron as a seperate weight for each arrow that goes into it. After finding a (the weighted sum passed through an equation), the neurn passes a on. Every arrow LEAVING that neuron contains the same value a.

I will add comments to the code to show what happens in each line.

Please tell me if this doesn't help, and which part(s) you didn't understand
Thanks for the comments ^^