Analysing new topics

Sun, 08 Sep 2002 22:58:49 +0100

I'm reading a paper [Joachims, 1996] about text analysis. The idea is to produce a facility within liveTopics for suggesting topics based upon the text entered in a post. At the moment there is a simple facility based upon a word search for existing topics, I'm keen to improve upon that in the future.

Does anyone know of any good papers on this subject?

Maybe I didn't need a network to predict this!

Thu, 24 Mar 2005 18:34:17 +0000

Neural networks are very cool but not suitable for all applications. I've basically been stumped by the problems of trying to use a network for indentifying interesting weblog posts.

The first problem is the input problem. How do you represent arbitrary chunks of text to the network in a meaningful way?

The problem here is that you have a layer of input neurons which form the input to the network. The inputs are driven by the environment (e.g. the text) and must consist of real values which can be fed to the next (hidden) layer of the network. If you're measuring temperatures, voltages, water levels, and so on then you are working with real values already. If you're working in image recognition you tend to have a fixed array of pixels (e.g. 640x480). But what about text?

I find myself presented with two sub-problems:

The length of the text is not finite. Measuring temperature you might have 2 or 3 sensors. A weblog post could as easily be 5 words or 50,000 words depending upon the authors whim. Although, in practice, you could say "no post will ever be more than 1MB of text" and treat 1MB as a limit that can create it's own problems.
Text doesn't neatly correspond to a real value input pattern. How do you represent the specific text as a numeric value?

An approach I formulated was to chop the input into keywords (doing appropriate stop-word rejection and so forth) and then feed the most relevant n keywords to the network as input for that text. The keywords could be uniquely numbered and then represented as a binary value. Each bit of the binary value would correspond to an input cell raising a value of 0.0 or 1.0 depending upon whether the bit is set or not. If we allowed a total of 4,096 possible keywords this can be represented in 12 bits (2^12=4096). If we used the 10 most relevant keywords for each post, i.e. n=10, then the input layer would, therefore, be composed of 12n or 120 cells.

However even having reached this point there are further problems to consider:

In a large post there may well be more than 10 keywords which means losing relevant information.
In a small post there may be less than 10 relevant keywords. What input is provided for non-existent keywords?
The relevance of the keyword to the item is not encoded. This problem might be solved by adding further input cells for each keyword to express keyword relevance.
The network acts as a feature detector. If we considering a set of temperature sensors wired to the inputs, each sensor will be wired to a specific set of inputs, they won't change. However a keyword that is detected in one position (i.e. represented in one set of input cells) for one item may be detected in another position for a different item and won't be considered the same feature. This is likely to be problematic.

Basically, when it comes to free text, input is a mess.

And then there are pragmatic problems like:

"How big should the hidden layer be?"
Too small and the network won't learn, too big and the network will mimic rather than learning to generalize properly and will also be slow.
"Should we have one hidden layer or two?"
As a general rule of thumb it appears that in 85% of cases 2 layers works best with 3 layers performing better in the rest.
"What training rate should be used?"
Set too high and the network bounces around unable to settle on a solution, set too low and it never converges on useful behaviour

Each of these problems is solvable but usually involves trial and error searching by training the network, deciding whether it is effective and, if not, junking it and trying some different combination of parameters. This is fine if you have a fixed training set with which you train the network repeatedly and then, having found the best parameter combination, just use it from there on.

However, in my application, the training is done by the user interactively and the training set will be different for each of them, and will change over time. Although the accumulated training data could be stored and the network retrained in the background I think this could mean that the network would never be useful.

All of which is leading me to think (as others have advised me) that a Neural Network may not be the best solution to this particular problem. I'm interested in whether anyone has any successful experience in this area. Otherwise I'm probably going to start looking more closely at Bayesian classifiers.

A cracking good read

Fri, 17 Jun 2005 23:53:52 +0100

Today I picked up a copy of Statistical Language Learning by Eugene Charniak. It seems like quite a thorough treatment of natural language processing using hidden markov models and probabilistic context free grammars. I'm also half way through Thomas Passin's Explorers Guide to the Semantic Web which is a very useful guide to the current state of the art in RDF w.r.t. agents, searching, logic, and ontology. I'm especially motivated by the relevance to the cognitive psychology reading I did on judgement, reasoning, and decision making this year.

Off the back of that I'm tinkering with a simple agent programming environment, written in Ruby, which I might release if it amounts to anything interesting.

Corpus

Sat, 15 Jul 2006 23:58:29 +0100

If anyone knows of a free modern English corpus that's available online I'd be very grateful to hear about it. Or if they can suggest an alternative way to solve my problem:

I have a database of terms that are used for making associations from English to a set of ideas. However because the number of terms relating to each idea may be different and because the popularity of each can be different it makes it very hard to compare how each is expressed.

It occurred to me that using a corpus I could build a frequency map to highlight popular/unpopular terms and apply discounts appropriately, normalizing (to some extent) the associations expressed and making comparisons more meaningful.

I am aware of the Brown corpus, the LOB, and the BNC. However each of these costs money which I don't have.

All ideas gratefully received.

Update: with Zed Shaw's help I found a page that includes a lexicon of about 57,000 English words with relative frequencies. I'm hopeful that this might offer a way forward.

Curiouser and Curiouser! on text-analysis

Analysing new topics

Maybe I didn't need a network to predict this!

A cracking good read

Corpus