Wsrtjtyk

MNIST is the hello world of machine learning and I've practiced it with TensorFlow and with pure python and numpy.

For more practice I am trying to write it in C on my own with only the standard library because I am relatively new to C and it's a great way to learn.

It's taken three weeks, and a lot of SEGFAULTS but I get 81% accuracy. Not very good but it's for learning.

The most troubling stuff was of course malloc/free for the data in the matrix struct as below:

typedef struct matrix{

    int rows, cols;

    float *data;

} matrix;

The forward and backward passes have things like:

1) matrix dot product

2) matrix add

3) matrix subtract

4) activation function (sigmoid in this case)

To avoid memory leaks I pass in three structs like so:

void matrix_add(matrix *a, matrix *b, matrix *res);

If res requires a dimensions change from a previous layer, then I free it and do a new malloc like so:

void zero_out_data(matrix *res, int rows, int cols)

{

  if (res->rows != rows || res->cols != cols)

    {

      if ((res->rows*res->cols) != (rows*cols))

    {

      free(res->data);

      res->data = NULL;

      free(res);

      res = NULL;

      res = malloc(sizeof(matrix));

      // make_matrix will calloc the data based on rows*cols

      // any other init stuff that could be needed

      make_matrix(res, rows, cols);

    }

      res->rows = rows;

      res->cols = cols;

    }

  else {

    res->rows = rows;

    res->cols = cols;

    for (int i =0; i < (rows*cols); i++)

      {

    res->data[i] = 0.0;

      }

  }

}

Then I can use that like so:

void sigmoid(matrix *z, matrix *res)

{

  zero_out_data(res, z->rows, z->cols); 

  for (int i = 0; i < (z->rows*z->cols); i++)

    {

      res->data[i] = 1.0/(1.0+exp(-z->data[i]));

    }

}

This gets very messy because a single forward pass has the following:

/* forward pass */

for (int k=0; k < (network->num_layers-1); k++)

  {

    matrix_dot(network->weights[k], activation, dot);

    matrix_add(dot, network->biases[k], zs[k]);

    sigmoid(zs[k], activation);

    sigmoid(zs[k], activations[k+1]);

}

/* end forward pass */

As you can imagine the backprop gets alot messier. I have to pre-create 8 different matrices, plus many more of those pointers to pointers of matrices like the activations and zs above, for the gradient descent.

What I would like to be able to do is return a matrix from a function like matrix_dot so that I can do:

sigmoid(matrix_add(matrix_dot(network->weights[k], activation), network->biases[k]));

That's kind of in the style of python/numpy.

Of course I can't return a local variable from a function because it's taken off the stack once the function returns.

If I return a pointer, then the above style will cause sever memory leaks.

Please note: I am not trying to write my own library/framework. I am simply trying to learn neural networks and coding in C. I have been a python developer for 7 years or so, and my C skills need improvement.

Memory leak in void zero_out_data(matrix *res, int rows, int cols)

matrix *res malloc out of the function and pass to zero_out_data. In zero_out_data, res is free and malloc again. If you want to change pointer res's value, then you need parameter like matrix **res.

If you want zero out data, no need malloc new matrix, just malloc the data part. I think your make_matrix function can malloc memory for data.

void zero_out_data(matrix *res, int rows, int col) {

      if (res->data == NULL) {

           make_matrix(res, rows, cols);

      } else if (res->rows != rows || res->cols != cols) {

          if ((res->rows*res->cols) != (rows*cols))

             {

               free(res->data);

               res->data = NULL;

               make_matrix(res, rows, cols);

             }

      }

      res->rows = rows;

      res->cols = cols;

      for (int i =0; i < (rows*cols); i++)

      {

            res->data[i] = 0.0;

      } 

}

How to implement this: sigmoid(matrix_add(matrix_dot(network->weights[k], activation), network->biases[k])); ?

You can use static or global variables to implement what you want. This will not be thread safe and reentrant. Examples in below:

matrix *matrix_dot(matrix *in_a, matrix *in_b)

{

        static matrix res = {0, 0, NULL};  // static variable



        // calculate the res's cols and rows number

        zero_out_data(&res, res_cols, res_rows);    // malloc new data



        // do some math.



        return &res;

}



// matrix_add will be just like matrix_dot



// I was wrong about sigmod no need new matrix. sigmod can also do it like matrix_dot.

You can use global variable replace static variable.

If you want thread-safe or reentrant, then just use local variable, then you can do it like this.

matrix *matrix_dot(matrix *in_a, matrix *in_b, matrix *res) 

{

      zero_out_data(res, xxx, xxx);

      // do some math

      return res;

}

// matrix_add will be the same.



// define local variables.

matrix add_res, dot_res, sig_res;

add_res->data = NULL;

dot_res->data = NULL;

sig_res->data = NULL;



sigmod(matrix_add(matrix_dot(network->weights[k], activation, &dot_res), network->biases[k], &add_res),  &sig_res)



// Now remember to free data in matrix

free(add_res->data);

free(dot_res->data);

free(sig_res->data);