Jeff Hawkins recently sent out a message via the Numenta newsletter describing their latest breakthroughs (his word, but i’m inclined to acquiesce for the time being – the work sounds very promising). FDR (for Fixed-sparsity Distributed Representations) is really only a means by which to represent input patterns. All the stuff about storage and associations and predictions and actions is separate. But FDR – if it works – is akin to zeros and ones in computers; it represents the lowest common denominator of means by which information is stored in a brain. (I won’t get into the details. A reader already posted this same link, but for convenience here it is again: http://www.youtube.com/watch?v=TDzr0_fbnVk
Hawkins make a reference to the “vast number” of combinations that can be stored in a binary array of 10,000 elements where something like 1000 of them turn on to represent a given state. I was curious about just how vast it was. Mathematically, the formula jargon is “10,000 choose 1000″, which means “how many distinct ways can 1000 elements be chosen out of 10,000″. Anyone who suffered through first year stats knows the answer is 10,000! / (1000! * 9000!), which in decimal is an integer with 1410 digits (which i think is all anyone needs to know about the number, except maybe that the first digit is not zero). So, “vast” seems an appropriate enough description. Of course, you’ll throw away a few orders of magnitude to accommodate for noise, but by my definition vast / big is still vast.
One of the things that Hawkins mentioned (perhaps casually) about this, though, was that it was important that any pattern have around 1000 elements; hence the “fixed” part of FDR. This becomes evident in the extreme if you think about a pattern with one element: you can only store 10,000 of them. Even if we generously call that number 5 digits, that’s still quite a bit less than 1410. Clearly, an input pattern of one element should somehow be “grossed up” by some reproducible means such that it becomes a pattern of around 1000.
Thinking more about this, it occurred to me that there is biological support for this process. If your back is touched with a fine point, chances are good only a small number of sensors will fire (as opposed to your finger tip or your tongue, where sensors are much more numerous). Even still, as the signals pass through the “waystations” in your spine, they are multiplied out so that by the time they reach cortex the signal is rich and complex. I can’t really elaborate on the subject at the moment – it was a while ago when i read about it in Kandel’s text – but i recall that there were a few explanations that were attempted to explain why the spine would take a perfectly good and clean signal and muck it all up. None of them were FDR.