Register    Login    Forum    Search    FAQ

Board index » FANN » General Discussion




Post new topic Reply to topic  [ 24 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Discrete-Time Recurrent Networks (GSoC)
 Post Posted: Thu Apr 12, 2007 1:03 am 
Offline
Site Admin
User avatar

Joined: Tue Mar 06, 2007 10:03 pm
Posts: 134
Location: Corn Desert (IL, USA)
Daniel Wilhelm's proposal has been accepted into the Google Summer of Code. Congratulations Daniel!

I've started this thread so others can come in and comment on the status of the project. To receive automatic email updates, look at the bottom of this page for the "Watch this topic for replies" link. I hope everyone can find the time stop by and give some encouragement, advice, or other support. Here is the body of his project application, and our conversation up until now:

Abstract
Recurrent networks are an important feature currently missing from the Fast Artificial Neural Network (FANN) library. Not only can they be used to model new problems, but they also better mimic the connectivity of biological neurons. For this project, the FANN library will be extended modularly to add support for discrete-time recurrent networks. The analogues of the feedforward training algorithms which FANN already supports -- recurrent backpropagation and recurrent cascade-correlation -- will be implemented. Unique to recurrent networks, Long Short-Term Memory will also be implemented to provide a method which can learn to store information over long time periods faster than recurrent backpropagation. This solution will include documentation, test cases, and a tutorial.

Detailed Description
===========================
GENERAL OBJECTIVES
===========================
This proposal focuses on quality rather than questionable quantity. First, it will provide a complete, functional, out-of-the-box solution. Not only will functions be written, but the corresponding documentation, test cases, examples, and tutorial will also be submitted. The additional code will be very modular, interacting with and requiring changes to existing code as infrequently as possible. Finally, the implementation will complement existing functions to minimize confusion. Most items implemented will be the recurrent analogues of preexisting feedforward algorithms, providing a very clean, easy parallel for new users.


===========================
FEATURES AND MILESTONES
===========================
(1) June 18 -- Recurrency functions will be added as an independent module. The 'fann' and 'fann_neuron' structures will be updated to allow recurrent connections by leaving the layered variables alone and adding additional variables to store additional connections. Separate recurrent network creation functions will be added which allow new networks to be declared explicitly as feedforward or recurrent. Functions which evaluate the outputs of recurrent networks will be added.

Effort will be made to allow recurrent networks to co-operate with existing feedforward creation routines by also accepting connections from the layer variables. This ability may also facilitate the addition of hybrid algorithms, or feedback networks which mostly utilize a feedforward structure. A classic, simple recurrent backpropagation algorithm will also be implemented -- recurrent back-propagation (c.f. Pineda, Almeida, Rohwer and Forrest). Note that this is the analogue of backpropagation for feedforward networks.

(2) July 9 -- As a parallel to the existing cascade-correlation algorithms for feedforward networks, the recurrent cascade-correlation architecture (c.f. Fahlman) will be implemented. Initially, because the general principles are the same, the implementation will be written by modifying the existing cascade-correlation code, significantly modifying it also to accept non-layers while retaining its existing functionality. Once complete, the code will be analyzed for performance, then it may be modified to exclude feed-forward networks if necessary.

(3) August 10 -- To extend the library's recurrent networks from backpropagation and cascade-correlation, a new architecture, Long Short-Term Memory (LSTM) will be implemented (c.f. Hochreiter and Schmidhuber, 1997). LSTM was developed because it is very slow to learn items for extended times by recurrent backpropagation. LSTM is a gradient-based method which can solve problems unsolvable by other techniques. The original paper has been cited ~200 times. An independent module which utilizes the recurrent routines implemented so far will be developed which correctly implements LSTM.

(4) August 20 -- Documentation on added functions will be clarified and added to the main manual. A tutorial article will be written for the new features. Examples written while testing will be added.

(5) If time remains, I will also individually update the bindings to accommodate for the added functionality.


===========================
DELIVERABLES
===========================
(1)
(a) Functions which initialize, modify, evaluate, and destroy recurrent networks. These functions will also utilize feedforward layers if the user creates them using the feedforward functions.
(b) A function which correctly implements the recurrent back-propagation algorithm.
(c) Test cases (with corresponding code) and basic documentation for all new functions.

(2)
(a) Implementation of recurrent cascade-correlation architecture.
(b) Test cases and basic documentation.

(3)
(a) Long Short-Term Memory implementation allowing users to create and train LSTM networks.
(b) Test cases and basic documentation.

(4)
(a) Additions to the manual of all user-accessible functions, a tutorial article, and example C files.

(5)
(a) Bindings which will work for given languages, the number of which depend on the amount of remaining time available.


Public Comment History
03/27/07 06:03
Seth J Price
Quote:
I like your application, but I have a few more questions for you:

I'm worried that the addition of many algorithms will make it difficult for beginners to use FANN.
- What changes do you expect to make to the external functions and enumerations of the FANN library?
- How will you document these changes?
- How will a novice user know if each algorithm is appropriate to be used for their problem? Create a short list explaining when it would be appropriate to use each algorithm. (Assume the user has a background in only basic neural net theory.)

How will you be sure that you have coded a correct implementation of these algorithms? Will you check your nets against MATLAB? (Is it already implemented in MATLAB?) Will you make a small regression test shell script that we can run? What about speed? How will you know that you've made an optimal set of functions?

How similar do you expect fann_create_shortcut() and fann_create_recurrent() to be?

03/27/07 08:03
Daniel Wilhelm (Student)
Quote:
Hi Seth. Thank you for the clarification requests.

(1) All submissions are identical.

(2)
>> "I'm worried that the addition of many algorithms will make it difficult for beginners to use FANN."

As mentioned, one of my largest goals is to make the transition from feedforward to recurrent easy for beginners:

(a) It will be easy to move from feedforward to recurrent since I will offer the recurrent analogues of familiar functions -- recurrent backprop and recurrent cascade-correlation. These will make the library have a very consistent feel, and it will make it easy to choose algorithms based on feedforward knowledge.
(b) I have incorporated time in the schedule exclusively to write a recurrency tutorial, clean up documentation and examples, and integrate them into the official manual, all of which will benefit beginners.


(3)
>> "What changes do you expect to make to the external functions and enumerations of the FANN library?"

This is included in my proposal in more detail. I hope to use the core FANN structures "fann" and "fann_neuron" since they are required for compatibility with existing functions. Unfortunately, they are at present not compatible with recurrent networks. Instead of removing the existing support for layers, I will add additional variables (e.g. a linked list) to store additional connections (the weight matrix can be simply expanded). Of course, existing functions will not operate properly with a recurrent network, so I will discuss with the project leaders the best trade-off of speed and user friendliness to prevent users from calling feedforward functions with a recurrent network.

The enumerations will require minor changes. One new enumeration distinguishing between a recurrent or feedforward network is necessary. In addition, since the large variety of backprop algorithms which are already defined are not yet available for recurrent networks, it is best to add the recurrent backprop training algorithm to the current list. Cascade support will be added as the current cascade support is -- as a completely independent module which is perhaps merged with the current one (see the proposal).


(4)
>> "How will you document these changes?"

See 2b above or my application. Of course, I will ensure that my code documentation is also consistent with the present FANN code documentation practices (or standards, should they be stricter!).


(5)
>> "How will a novice user know if each algorithm is appropriate to be used for their problem?"

Luckily, this decision is easy. Recurrent vs. feedforward will be discussed in the tutorial, and the second decision a user must make (backprop vs. cascade) is completely analogous to the preexisting feedforward implementations. Because recurrent networks are bidirectional, they can much better model temporal tendencies. Hence, adaptive architectures and control systems which change over time are great applications. In addition, recurrent networks often provide faster convergence times or require a smaller number of neurons for traditional problems. Finding good neural network configurations is often considered an art, so users may want to try their problems with recurrent networks if they are not able to meet their desired constraints with feedforward.

If the network is decided to be recurrent, then the user must only decide between backprop and cascade correlation. If the user has a certain structure in mind, he or she can use the backprop algorithm. Otherwise, the user could try the cascade correlation method which will automatically build and train the network. Really, it may be desirable to try both methods so the user can compare the accuracy and time required for training.

Finally, LSTM is an architecture appropriate for many uses. It is most ideal when the data is temporal and long time lags exist between relevant events. Its second strength is being able to learn quickly compared to other networks such as the recurrent implementation. The main downside is that LSTM imposes a network structure which may require more nodes than optimal.


(6)
>> "How will you be sure that you have coded a correct implementation of these algorithms?"

To ensure correctness, a number of test cases will be generated, including possible corner cases. These will be dually run through my code and simulators which others have written. Iterations to convergence, the structure of the resulting network, and final weights will all be considered. I will definitely provide you with a mechanism for regression checking, since I will write scripts anyway which will run my test cases through my software and theirs. I will also retain the version of their software and the test cases I used.

The recurrent cascade correlation implementation will be tested against Fahlman's own simulator. The LSTM implementation will be tested against Hochreiter's own simulator. Recurrent backprop has been widely implemented (likely also in Matlab's NN Toolbox), so it may be tested against several implementations.

A dual MATLAB implementation will be written if I am stuck on the C implementation. Since simulators are available from the technique authors, hopefully I will be able to resolve most mistakes by understanding their papers and implementation thoroughly before writing my own. As can be expected, no plagiarism will occur.


(7)
>> "What about speed? How will you know that you've made an optimal set of functions?"

Regarding speed, I have been writing highly optimized C code for years. However, my policy is to get an implementation working before optimizing it. Once it works, if time permits I will optimize it using profiling and/or my own hunches!

I am not sure what is meant by an "optimal set of functions", but my main criterion is to ensure that the functions are easy to use by newcomers and powerful enough for experts.


(8)
>> "How similar do you expect fann_create_shortcut() and fann_create_recurrent() to be?"

There will likely be a small number of initialization functions which cater to the needs and experience of the user. A completely recurrent network would only require a single parameter, the number of neurons. A feedforward-like recurrent network would be very similar to the fann_create_shortcut() call (with an additional parameter distinguishing the degree of recurrency). An entirely recurrent network would require a matrix of which neurons are connected to which other neurons.

We can discuss the best combination, but I foresee these three being important for complete beginners, users familiar only with feedforward networks, and advanced users who want full control!

-------------------------------------------------------

As an additional note (as if this hasn't been long enough!), I am flexible on the third LSTM portion of the proposal -- if another student will be working on a great Kohonen map implementation, it would be absolutely super to also have a (less extensive) *recurrent* self-organizing map learning algorithm which would take advantage of the recurrency features just added (but without all of the bells and whistles that someone working on Kohonen maps full-time would implement). So, implementing just the training algorithm for this by using the recurrent network structures I developed during the first two-thirds would also be an exciting possibility (instead of LSTM).

Thank you for the consideration!


Top 
 Profile  
 
 Post subject:
 Post Posted: Thu Apr 12, 2007 3:12 am 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
Hi everyone, I'm Dan!

I look forward to working with the FANN community and meeting the other students -- this will indeed be a summer of code! I'm hoping to keep you all up-to-speed with my progress via a thread in these forums, sort of like a blog. And, I'll be in touch with my mentors through IM.

I'm certainly excited about recurrent networks, but after seeing the other great proposals, my interests have suddenly diversified :-) If everything works to plan, SoC could really propel FANN to the next level, both in usability and in content.

Just some quick background info -- I'm a graduate student at Caltech in their Computation and Neural Systems PhD program. I've been interested in neural networks for some time and have even used FANN for a school project in the past. I see recurrency as perhaps playing a vital role in my thesis work, and hence working on a project such as this will be great both for FANN and for my own enrichment :-)

I can't wait to meet the rest of you -- thanks for having confidence in my application!

_________________
Dan


Top 
 Profile  
 
 Post subject:
 Post Posted: Sat Jun 09, 2007 10:56 am 
Offline
User avatar

Joined: Tue Mar 06, 2007 7:24 pm
Posts: 264
Location: Copenhagen, Denmark
Hi Dan,

How is the project going?

As noted on the sticky GSoC thread, all students are required to give some kind of status each week. You have failed to do so, please let me know how your project is going.

Best Regards,
Steffen

_________________
Steffen Nissen - http://facebook.com/profile.php?id=595485027
Project Administrator - Fast Artificial Neural Network Library (FANN)
http://leenissen.dk/fann/


Top 
 Profile  
 
 Post subject:
 Post Posted: Sat Jun 09, 2007 9:04 pm 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
Thank you for the reminder, Steffen!

(1) Part of my first milestone was to implement a mechanism for storing recurrent networks within FANN. The MIMO implementation supports this, and I am closely following its implementation via hawk's repository. I will provide additional comments via that thread. In the meantime, with the provisionary source from his repository, I am implementing the recurrent backpropagation algorithm from the following paper:

Pearlmutter, "Learning state space trajectories in recurrent neural networks"
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=118724

I am also considering implementing some improvements, such as:

Fang, Sejnowski, "Faster learning for dynamic recurrent backpropagation"
http://papers.cnl.salk.edu/PDFs/Faster%20Learning%20for%20Dynamic%20Recurrent%20Backpropagation%201990-3145.pdf


(2) Because hawk is implementing MIMO, I am implementing the discussed parameter structure from the parameter discussion thread -- http://leenissen.dk/fann/forum/viewtopic.php?t=110. This is a very straightforward change, but a crucial update to avoid polluting the primary FANN structure even more than it currently is. The bulk of the work in this update is actually changing variable accesses to access the new sub-structures. (Speaking of which, should we call the new get/set functions even from within our own code to access these?)

The important part is that with this change, users will not need to update any code at all -- this is a purely internal operation. I am over halfway complete, and I will commit this once it is finished and tested.


(3) I will also post a thread shortly discussing another very important issue -- regression testing. We need an easy-to-use, consistent system for writing test cases, automatically building them, and comparing their output to the desired output. We will all generate test cases to ensure our new code is working, so if we can save these and get them into the regression system, then that will work wonders for future builds.

_________________
Dan


Top 
 Profile  
 
 Post subject:
 Post Posted: Sun Jun 10, 2007 8:50 am 
Offline
User avatar

Joined: Tue Mar 06, 2007 7:24 pm
Posts: 264
Location: Copenhagen, Denmark
I am glad to hear from you, you address two important issues here, that I think deserves a separate thread, and a mention on the sticky GSoC thread.

1) The issue of whether the library internally should use it's own get and set functions. This has not been the style up until now, but my personal opinion is actually that the get and set functions should be used whenever possible, which also means in the internal library code, as long as it does not hurt performance.

2) The issue of testing. There exists a benchmark framework, which will do the trick for testing speed, and performance of new algorithms. I think that this framework could be tweaked to do a bit more testing, but other internal unit-tests would also be a great idea.

_________________
Steffen Nissen - http://facebook.com/profile.php?id=595485027
Project Administrator - Fast Artificial Neural Network Library (FANN)
http://leenissen.dk/fann/


Top 
 Profile  
 
 Post subject:
 Post Posted: Sun Jun 10, 2007 3:18 pm 
Offline
Site Admin
User avatar

Joined: Tue Mar 06, 2007 10:03 pm
Posts: 134
Location: Corn Desert (IL, USA)
I agree that internally we should still use the higher level functions. It will make the library more resilient to changes in the code, which is something that will happen often this summer. In fact, I would go so far as to say always use the higher level constructs (when possible), unless you profile the code and see that the function call slows the library down.


Top 
 Profile  
 
 Post subject:
 Post Posted: Sat Jun 16, 2007 8:16 am 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
For this weekly update:

+ Last week, I implemented and committed the parameter scheme that was discussed in the parameter pollution thread: http://leenissen.dk/fann/forum/viewtopic.php?t=110&start=15. There are still some GET/SET functions which need to be added, but this will be a quick addition.

+ Over the next 2 weeks, I will get a recurrent network backpropagation algorithm working with the current MIMO implementation. When MIMO is checked in to FANN (or shortly afterward), I hope to check the first of the recurrent algorithms in.

+ I am moving places of residence this weekend, and I will not have Internet access until early next week.

+ I have been wanting to research more into automated regression testing, but I have been busy preparing for my move. I will post a thread about this in this coming week for sure with some proposed options since this is a critical topic.

_________________
Dan


Top 
 Profile  
 
 Post subject:
 Post Posted: Mon Jun 25, 2007 8:01 am 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
+ I will commit the Check tests in the next few days. The new files will not require any changes to the FANN code -- they are in addition to it only. I'm doing these odd jobs (i.e. GET/SET) since my project is highly dependent on the internal representation of the network -- MIMO.

+ I have the recurrent backprop written according to the MIMO code in hawk's GIT repository. Unfortunately, I believe the repository is missing at least one file, and so I will test the code with hawk's when I can get this resolved. Then, I'll upload this code along with MIMO when it is committed.

+ I'd like to remove the internal GET/SET calls. They are VERY long and make the code look ugly. Not only this, many SETs (e.g. to arrays) cannot be done using the GET/SET methods, and so currently the converted file is a mix-mash of GET/SETs and classic struct accesses. (I only converted a single file due to these problems.)

+ MIMO requires a "config.h" generated by ./configure. If possible, I'm going to update the MSVS project files and src to be compilable by MSVS (and GCC).

_________________
Dan


Top 
 Profile  
 
 Post subject:
 Post Posted: Mon Jul 09, 2007 7:41 am 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
Summary of first-half achievements. Hawk committed to a MIMO implementation which would allow for recurrent networks (part of my first milestone), so I concentrated on more general improvements to the library in the first half; an internal representation of a recurrent net is necessary for implementing recurrent algorithms!

1. Parameter clean-up. This was performed to reduce pollution of the "fann" structure and to increase organization. More information can be found in this thread: http://leenissen.dk/fann/forum/viewtopic.php?t=110. The changes were committed to the gsoc repository. No changes to the user interface were made -- this was a purely internal change.

2. Regression testing. I incorporated the CUnit unit/regression testing (http://cunit.sourceforge.net/) into FANN. More information can be found in this thread: http://leenissen.dk/fann/forum/viewtopic.php?t=145. I added the "/tests" directory, containing an initial five tests based on the examples and code to run the regression tests. These files were committed, and a Visual Studio .NET project file was committed. I'm having problems auto-tooling the gsoc repository (but not hawk's Git one), so I'll wait until hawk's MIMO is committed before changing the Makefile, unless I am able to solve the problem earlier.

3. Recurrent algorithms. I wrote and committed a new create ann routine, create_unrolled_recurrent (fann_recurrent.c), and a CUnit test for it (fann_unrolled_recurrent.c). This function converts any recurrent network (given by weight matrix) into a feedforward network which precisely evaluates the recurrent network after a certain number of time steps. I also implemented Pineda's "Generalization of backpropagation to recurrent neural networks." (See http://prola.aps.org/abstract/PRL/v59/i19/p2229_1.) This algorithm generalizes backprop to allow training of recurrent networks, given that the network has a fixed point (i.e. becomes stable). This latter algorithm will be committed to the repository in the next few days, but it is only compilable with the MIMO implementation.


Coming soon:

The second half of the GSOC program will be used to implement some of the most popular and well-accepted recurrent learning algorithms. These include Back Propagation Through Time and Real Time Recurrent Learning. These algorithms are especially exciting because they allow for the recognition of temporal sequences of data, something that FANN currently lacks. These algorithms are being implemented using the in-progress MIMO code.

_________________
Dan


Top 
 Profile  
 
 Post subject:
 Post Posted: Fri Jul 13, 2007 6:22 am 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
psyklic wrote:
Pineda's algorithm will be committed to the repository in the next few days


I discovered a bug, but I will have this recurrent algorithm up soon.

_________________
Dan


Top 
 Profile  
 
 Post subject:
 Post Posted: Mon Aug 20, 2007 5:43 pm 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
I did not finish my original goals, but following are my accomplishments so far:

Fully Recurrent MIMO Networks. Fully recurrent networks are implemented as a single layer containing a single MIMO neuron with 'n' outputs, corresponding to the 'n' recurrent neurons of the network. The network also contains 'm+1' input lines (or input nodes), of which 'm' are fixed to input values and one bias line is fixed to the value '1'. Each input line connects with each neuron unidirectionally. Each recurrent neuron connects to each other recurrent neuron bidirectionally. The last 'o' nodes of the recurrent neurons are designated as output nodes. An example was included as a Hopfield network.

Fully Connected Shortcut MIMO Networks. Using MIMO, this is implemented as follows. A single contiguous block of memory for storing *all* outputs is initially allocated. Then, this is partitioned between layers and neurons. The key is that the lowest memory region contains the inputs, and higher memory regions contain larger-numbered layers. Hence, by storing the number of inputs per layer and a pointer to the beginning of this block, no memory is wasted by duplicating outputs, and all layers can share the same set. An example was included.

Hopfield Network. Autoassociative networks are fully recurrent. However, instead of there existing 'm' input lines, the 'n' nodes are directly initialized with values at time 0. Hence, autoassociative networks can be created using the recurrent network architecture by creating it with zero inputs and 'n' outputs. Then, before running the network, each node must be filled with a pattern. Autoassociative networks are often used for recalling patterns from partial data. Included an example file with create/train/and run routines.

"Unrolled" Recurrent Networks. As Rumelhart and McClelland used for backpropagation through time, any recurrent network can be expanded out for some 't' time steps and simulated using standard feedforward running and learning rules. This routine converted any recurrent network into a 't' layer feedforward network.

Everything listed in my midterm posting.


I will continue to develop recurrent routines for FANN, most pertinently:

Completing the Real-Time Recurrent Learning algorithm. I have an implementation uploaded and an example file, but the MSE does not seem to readily decrease. I'm close though, so I will finish it!

Updating the Cascade Correlation algorithm to MIMO. Then, the Recurrent CC algorithm can be implemented.

Restoring the above algorithms. I developed all of the above in Visual Studio since you all know that I'm not a big Linux user. I couldn't figure out how to compile my new files with CMake, however, so I decided to remove the build_cmake directory and start over, like so: "$ rm build-cmake/ -R *" which, unlike DOS, does not only remove that directory, but everything in my current path, including the source code. Luckily, I still had two files open in Visual Studio which were saved, and so I uploaded them to 'recurrent_remains' in /src (in addition to some documentation I was writing). I will rewrite the files and extend them as mentioned, but since I can't produce adequate source code I suppose that I am out of the Google program.

I look forward to continuing to work with you all on FANN, however! I apologize for my naivete.

_________________
Dan


Top 
 Profile  
 
 Post subject:
 Post Posted: Tue Aug 21, 2007 5:47 am 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
This morning, I uploaded the code which was serendipitously saved by Visual Studio. I hope to rewrite my fully recurrent and shortcut MIMO code as soon as possible so that my implementation is complete. I feel terrible having worked for a long time on the code only to mess up when trying to erase a directory.

It's been a pleasure working with many of the other GSoC students and mentors. Although I wish we could have interacted more frequently, the experience was wonderful, teaching me about open source development, neural networks, and working as part of a team. I hope that FANN continues to be supported by Google!

_________________
Dan


Top 
 Profile  
 
 Post subject:
 Post Posted: Mon Aug 27, 2007 7:57 pm 
Offline
User avatar

Joined: Tue Mar 06, 2007 7:24 pm
Posts: 264
Location: Copenhagen, Denmark
Hi Dan,

What is your status?

Best Regards,
Steffen

_________________
Steffen Nissen - http://facebook.com/profile.php?id=595485027
Project Administrator - Fast Artificial Neural Network Library (FANN)
http://leenissen.dk/fann/


Top 
 Profile  
 
 Post subject:
 Post Posted: Thu Aug 30, 2007 4:06 am 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
At long last, I have finished restoring the recurrent files. Thank you all for your patience! The files implementing recurrent networks include fann_recurrent.c, fann_recurrent.h, and fann_basic_fully_recurrent.c. There have also been a few additions to other files to better support these networks.


I have included four easy-to-follow examples which showcase all of the recurrent functions, and I have just verified that they all compile and run. These examples were written to verify that the functions work, and you can clearly see that they do by running these and verifying the answers manually (if you wish). Hence, these will make great CUnit test cases! (Find them in the /examples directory.)

recurrent_basic_test.c - Creates a random recurrent network, displays the connections, and displays parameters.
recurrent_unrolled_test.c - Creates an unrolled recurrent network out to a random time era. Any recurrent network can be converted to a feedforward network (for a given time step), and this test performs this! Note that the final few (random) neurons are the biases, and so they do not have a repeating pattern.
recurrent_hopfield_test.c - Here, we create a recurrent Hopfield network, train it, and show that it works via two input files and random test cases.
recurrent_rtrl_test.c - Here, we first verify that we can run the network forward one time step, then we verify that MSEs are computed correctly. Finally (requiring enabling) you can see the in-progress RTRL algorithm.

I have submitted detailed documentation regarding the internal layout of the recurrent networks in the /doc directory. I actually had a chance on this rewrite to make a better internal structure, and so I have implemented that this round. I believe that the internal structure is rather elegant, using layer->inputs/outputs for the total inputs/outputs to the layer (i.e. inputs being all inputs + bias, and outputs being all the neurons in the layer) whereas the ann->inputs/outputs are the total from the network itself (i.e. the input lines and only those neurons which output values). I implemented the "input line" structure, so the "input layer" is now not a separate layer, but those inputs are fully connected to all other neurons regardless.



I have determined that most of the problems/frustrations I've had this summer stem from my choice of compilers and this choice requiring that I have an intimate knowledge of some of the new codes. You see, I decided to use Cygwin and Visual Studio, both of which have incompatibilities with straight-up Linux compilers. Hence, every time I downloaded new source I had to reconfigure it so that I could use Visual Studio and then Cygwin. So, I had to understand your new code as you uploaded it, which was mildly frustrating, but fruitful. Here are some workarounds I had to use for this final build -- I did not want to mess up anyone's hard work:

+ I could not get dlfcn to work in Cygwin, and it is downright not supported in Visual Studio without a third-party library. The (current) workaround for this is to manually insert the constructor function pointer yourself.
+ I had one particularly hard to track bug where the library was built with one #include "fann.h" and the examples (in the examples directory) actually referred to a second #include "fann.h" in which 'struct fann' was defined differently! Symptoms of this include stack bounds errors and values being modified in different regions of the 'fann' structure than intended. The workaround was to #include "../src/include/fann.h" in the examples directory so that I could be explicit about the path.


Enjoy!

_________________
Dan


Top 
 Profile  
 
 Post subject:
 Post Posted: Thu Aug 30, 2007 8:13 pm 
Offline

Joined: Thu Apr 12, 2007 2:52 am
Posts: 88
I realized that I forgot to commit two test pattern files for the Hopfield example, and so they are now in the /examples directory.

_________________
Dan


Top 
 Profile  
 
Display posts from previous:  Sort by  
 
Post new topic Reply to topic  [ 24 posts ]  Go to page 1, 2  Next

Board index » FANN » General Discussion


Who is online

Users browsing this forum: Google [Bot] and 0 guests

 
 

 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: