New Video Tutorial: Make a Neural Net Simulator in C++

February 4, 2011

Released today! Our newest video programming tutorial, A Neural Net Simulator in C++, is now available for your viewing pleasure at the following link:

[ Update for 2013: Also see the new companion video for visualizations of how neural nets work and how to train them: The Care and Training of Your Backpropagation Neural Net.  ]

Neural Net in C++ Tutorial from David Miller on Vimeo.

If you’re a beginning to intermediate C++ programmer, this tutorial will guide you through the analysis, design, and coding of a command line console program that implements a neural net in C++. You’ll end up with a classic back propagation model with adjustable gradient descent learning and adjustable momentum. You’ll see how to teach your neural net to solve a simple task, then you can supply your own training data to train your net to do wonderful and amazing things.

Besides showing how a neural net works, we also discuss:

  • C++ class design
  • prototyping with portable C++
  • test early, test often
  • encapsulation, data hiding
  • static class members
  • accessor functions
  • const correctness
  • the assert() macro
  • the vector<> template container, and .size(), .back(), and .push_back() member functions
  • reference variables

This tutorial does not cover or require exception handling, memory management, STL iterators, inheritance, threads, or graphical input or output.

The finished neural net source code is also available for download here (see the video for instructions on what to do with it):

 

 

tags: , , , , ,
posted in C++, Tutorials by Dave

Follow comments via the RSS Feed | Leave a comment

18 Comments to "New Video Tutorial: Make a Neural Net Simulator in C++"

  1. Ben wrote:

    Hello,

    You have a bug in your code. After you set up the net in the Net() constructor, you do not initialize the bias node.

    This code must be slightly different than the code you showed in your online video, because the one in your online video has this initialization.

    Essentially ,you’re missing this line:

    // force the output node’s bias to 1.0
    m_layers.back().back().setOutputVal(1.0);

  2. Dave wrote:

    Ben, thank you very much for spotting and reporting that omission. The downloadable source code should match the video now. (For reference, the line we’re talking about is near the end of the video at around 59:45).

    The bias neurons are interesting. As far as I can tell, they don’t have to be initialized to any particular value; they just need to be nonzero. Then the backprop algorithm will change the connection weights of the bias nodes to scale the value to whatever the neurons need in the subsequent layer.

  3. Theo wrote:

    Hi Dave, first of all thanks for this great video tut. It really helped me to understand some certain things:)

    I downloaded the code for windows(dos version) and after building the project(which by the way ends without any errors or warnings), when I stard .exe file I get the message:

    Debug Error!
    R6010
    - ()abort has been called

    can you give me a hand on this issue?thanks in advance:)

  4. Dave wrote:

    Hi Theo, the only abort() statement in the program is at line 45, so this should be easy. It checks that the input training data begins with a line that contains the string “topology:”. So just be sure that your input training file is formatted the same way it is shown in the video and it should get past that abort(). The sample input file used in the video (at around 1:00:50) begins with these lines:


    topology: 2 4 1
    in: 1.0 0.0
    out: 1.0
    in: 1.0 1.0
    out: 0.0

    As mentioned in the video, that entire class called TrainingData was intended to be just a placeholder, with the expectation that you’ll replace it with something more appropriate for your needs, and so that class is not burdened with a lot of friendly error messages and error recovery. So if you use the TrainingData class as shown without modification, just be careful to supply it with the same format of input data used in the video.

  5. Theo wrote:

    Thanks a lot Dave:)I got it..

  6. B Rehart wrote:

    This is my favorite tutorial! It’s over an hour long and I didn’t even notice. I always wondered about neural nets and how they worked. Since it seems they take a lot of iterations to learn stuff, I wonder how to optimize them for game AI agents. AI agents can’t take 2000 iterations to learn not to attack an opponent through the valley, surrounded on both sides in the killing zone. Is there a faster learning algorithm?

  7. Dov wrote:

    This is a nice intro to neural networks, and it’s certainly making my task of getting something going in that area easier. However, you made one serious mistake. The rand function has no specified implementation, so it’s highly nonstandard. On Turbo C way back when, I actually saw an implementation that cycled after 137 numbers! Hopefully no implementation is as bad today, but rand() is not designed for floating point applications. drand48() is the ANSI standard uniformly distibuted u[0,1]. Even better, use boost, but if you’re trying to keep things simple I can understand why your wouldn’t.

    The other nitpicky things are not serious, but I don’t see why you abstracted certain things the way you did. Bias seems to me to be intrinsic to the neuron, so it seems to me that having a private member with the bias number is the way to go, not creating an extra bogus Neuron that is always one, and then weighting it. Sure it’s slower and bigger, but it’s not the speed, it’s the lack of clarity that bothers me. Similarly, I don’t see that you gain anything by making a bunch of bogus neurons on the input. The abstraction should be that you link each neuron with the location where its input values are.

    Any way you slice it, you’ve done a real service to anyone getting started in neural networking. I hope these comments serve to make your code even better.

    Dov

  8. Dov wrote:

    Oh, another small thing I noticed, you pass strings by value unnecessarily, and don’t always const parameters that are readonly:

    void showVectorVals(string label, vector &v)

    should be:

    void showVectorVals(const string& label, const vector &v)

  9. Dave wrote:

    Hi Dov, Thanks very much for the thoughtful feedback; it deserves a thoughtful reply.

    You’re absolutely correct that showVectorVals() could take its string argument by reference to const. That’s a good suggestion.

    The bias input is slightly special, but only because its input is constant when running a trained simulator. The reason the bias neuron is not a private member of the neuron class is so that the backpropagation loop can adjust its weight the same way it adjusts all the other weights. A neuron is just an integrator, and the bias input carries what you could think of as a DC component that goes with the signals that are on the other inputs. The bias is no more or less important than the other input signals that get summed, and its weighting is recalculated during training in the same way as the other neurons. They’re more alike than different, and many loops are simplified by including the bias neuron in the loops. But you can certainly abstract it differently and implement it in several alternative ways, all mathematically equivalent and all quite correct.

    I appreciate your comments about the deficiencies of rand(), but it’s not a problem here. Besides, that’s our only choice if we’re sticking with standard C++ features. (The drand48() function is not standard in C++98 nor C++11.) In C++-11, you could use new cryptographic quality random number generators if you needed random bits that would survive intense statistical analysis. But we don’t need that in this program. We’re only using rand() to initialize weights so that the infant neural net begins its walk toward a solution somewhere in the midst of the huge solution space instead of always starting out at the same remote corner of the solution space. We only need randomness like a shotgun. Even the old Turbo C implementation would have sufficed. And as you can see in http://en.wikipedia.org/wiki/Linear_congruential_generator#Parameters_in_common_use, the rand() function as implemented in today’s compilers is way more than sufficient for our needs.

    Thanks again for the comments!

  10. Steve(UK) wrote:

    Really enjoyed your tutorial David , helps in understanding the basics in Neural Networks and what a bonus some c++ coding to play around with, many many thanks.

    Best Regards
    Steve

  11. Irina (CR) wrote:

    This is the first time I see one of your tutorials. I have to say, I’ve been really interested in AI for some years now, but I’m still an undergraduate student just seeing where her career is going, which I hope is that area. I was looking for information about neural nets, and I found your video. This not only made me love more AI, but inspired me. Thank you for your awesome tutorial and for sharing your knowledge.

  12. Hindi wrote:

    Thanks a lot for this tutorial. I followed some courses about neural networks and leaving the theory for some practice is really interesting.

  13. CK wrote:

    When calculating gradients in hidden Layers, you calculated bias’s signal error. Is that true ?
    Bias is not fixed ?
    Thanks

  14. admin wrote:

    That’s right, during training, the biases gets adjusted during each backpropagation pass.

  15. Eak wrote:

    Greetings;
    when I tried your written c++ code for the Neural Net with visual studio 2012. when I debugged it I got the (R6010 Error: abort() has been called. the the program stops.
    thanx

  16. Dave wrote:

    Hi Eak, check out comment #4 above (http://www.millermattson.com/dave/?p=54#comment-75 ) — the only abort() in the program occurs if the input data file isn’t formatted as shown.

  17. Eak wrote:

    Greetings;
    Its me again… I got the (E6010 Abort() has been called) even when I try to run the makeTrainingSamples.ccp

  18. Dave wrote:

    Sorry you’re having troubles with it. You may need to run the neural net with a debugger and set a breakpoint somewhere in lines 41 – 44 and see why the string “topology” is not being found in the line that was read from the file.

Leave Your Comment

*
 
Powered by Wordpress and MySQL. Theme by Shlomi Noach, openark.org