Curiouser and Curiouser!

Curiouser and Curiouser! on bayesian-classification http://matt.blogs.it/ RSS feed for topic bayesian-classification Copyright 2006 Matt Mower Squib/0.1 self@mattmower.com self@mattmower.com en-gb K-Collector & Bayesian filtering Sun, 17 Aug 2003 18:59:05 +0100 <BLOCKQUOTE> <A href="http://www.decafbad.com/blog/geek/bayes_agg_one.phtml">Issues in using SpamBayes to filter news items</A>. Despite a reading <A href="http://www.srijith.net/trinetre/archives/2003/08/11/index.shtml#000373">an entry by Srijith</A> discussing Bayes-based classification as unsuitable for use in news aggregators, I tied <A href="http://www.spambayes.org">SpamBayes</A> into my <A href="http://www.decafbad.com/viewcvs.cgi/dbagg/">homebrew news aggregator</A> and have been trying it out this week. I know Ive been <A href="http://www.decafbad.com/blog/geek/syndicated_whuffie.phtml">talking about it</A> for awhile, but procrastination and being busy all round kept me from getting to it. Funny thing is, when I finally got a chance to really check things out, the integration was a snap. Id anticipated a bit of work, but was pleasantly surprised. I doubt that any other aggregator written in Python would have a hard time with it. If, that is, anyone else wants to do it. I already knew it wasnt <A href="http://diveintomark.org/archives/2003/07/07/linkdumps_are_like_sex">magic pixy dust</A> but I figured it might be worth a try. I will be eating my dogfood for awhile with this, but Im thinking already that whats good for spam might not be so good for news aggregators. Srijiths <A href="http://www.srijith.net/trinetre/archives/2003/08/11/index.shtml#000373">post</A> mentions some snags in ignoring some of the semantics of a news item, such as whether a word appears in the items title or information about the items source. I dont think that this completely applies to how Im doing classification, since SpamBayes appears to differentiate between words found in email headers and the body itself. When I feed an item to SpamBayes for training and scoring, I represent it as something like an email message, with headers like date, subject, from, and an X-Link header for the link. However, even with this, I think Srijiths got a point when he writes that this method will miss a lot of available clues for classification. Unlike Srijiths examples, though, Im not trying to train my aggregator to sift entries into any specific categories. So far, Ive been trying to get it to discriminate between what I really want to read, and what Im not so interested in. So, I figured that something which can learn the difference between spam and normal email could help. But, although its early, Im noticing a few things about the results and Ive had a few things occur to me. See, in the case of ham vs spam, I really want all the ham and none of the spam. A method to differentiate between these two should be optimized toward one answer or the other. SpamBayes offers I dont know as a third answer, but its not geared toward anything else in-between. However, in measuring something like interest, inbetween answers are useful. I want all of the interesting stuff, some of the sort-of interesting stuff, and a little of the rest. This is also a problem for me in deciding to what I should give a thumbs up and what gets the thumbs down. Even though Ive subscribed to a little over 300 feeds, every item from each of them is somewhat interesting to me. I wouldnt have subscribed to the feed if there wasnt anything of interest there, so Ive already biased the content of what I receive. Some items are more interesting than others, but the difference between them is nowhere near the difference of wanted ham vs unsolicited spam. So, I find myself giving the nod to lots of items, but only turning down a few. SpamBayes would like equal examples of both, if possible. Ill still be playing with this for awhile, but I need to look around at other machine learning tech. Im just hacking around, but the important thing is to try to understand the algorithms better and know how they work and why. Bayes is in vogue right now, but as Mark Pilgrim intimated, its not magic. Its just advanced :) In the immortal words of <A href="http://www.spidereyeballs.com/os6/set3/small_os6_d3_3596_sm.html">Mark Jason Dominus</A>: You cant just make shit up and expect the computer to know what you mean, retardo! [<A href="http://www.decafbad.com/blog">0xDECAFBAD</A>]</BLOCKQUOTE> Interesting.  Within <A href="http://w4.evectors.it/">k-collector</A> we already have a method for selecting a level of interest more granular than the feed and that is the topic (and, soon, groups of related topics). This allows you to say I'm interested in Java rather than I want to read these 200 blogs where they talk about Java sometimes.  Then within this view you could start to say well, so and so is more interesting than Matt on this topic.  But again, you are only dealing with the topic at hand.  You might still think I'm more interesting about something else even if I struggle for an example! We're already looking at interesting things we can do with this approach, maybe Bayesian filtering is something we should be thinking about. http://matt.blogs.it/entries/00001543.html Crime is bad Reverend Tue, 19 Aug 2003 10:01:29 +0100 <A href="http://osoft.us/Members/jeff/permanent/b-2003Feb22-104PM/book_view">Jeffrey Hicks</A> sent me a comment in response to my <A href="http://matt.blogs.it/2003/08/17.html#a1073">recent post</A> thinking about introducing Bayesian filtering into <A href="http://w4.evectors.it/">K-Collector</A>.  He pointed me at an application he is demoing called <A href="http://jrhicks.net/reverend">Reverend</A>.  It's combines a Bayesian filter with the <A href="http://www.cogsci.princeton.edu/~wn/index.shtml">WordNet</A> lexical classifier to allow people to train it about whether words are good or bad. The result is that it can tell you that crime is bad and, by inference, that bribery, racketeering, larceny and theft are bad too! Somewhat reminiscent of some of the <A href="http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=cyc">Cyc</A> inference examples. http://matt.blogs.it/entries/00001555.html Bayes makes topics Tue, 19 Aug 2003 19:57:08 +0100 Doing a bit of digging into Bayesian filtering.  Although it may have other uses in K-Collector later on, my initial thoughts are using a Bayesian filter to automatically suggest topics for weblog posts via K-Collector client.  If it's good enough we might even be able to skip the user having to approve some topics. At the moment the client is using a simple keyword stemmer which is effective, up to a point, but suggests a lot of false positives.   I'm hoping that a trained Bayesian classifier will do a lot better.  Of course this raises the issue of how it gets trained but that is a bridge of a very different colour. Some resources I have come across: <UL> <LI><A href="http://www.paulgraham.com/spam.html">A Plan for Spam <Paul Graham></A></LI> <LI><A href="http://sourceforge.net/docman/display_doc.php?docid=13334&group_id=63137">Project POPFile</A></LI></UL> http://matt.blogs.it/entries/00001558.html Experienced classifier wanted Tue, 19 Aug 2003 20:18:41 +0100 One question bubbles up in my mind reading about Bayesian classifiers.  They all seem to be naive.  So, what does an experienced Bayesian classifier look like? http://matt.blogs.it/entries/00001565.html NewsBayes Thu, 20 Nov 2003 16:23:49 +0000 <div style="margin-left: 40px;"><a href="http://weblog.infoworld.com/udell/2003/11/20.html#a851">Working with Bayesian categorizers</a>. </div> <blockquote cite="O'Reilly Network" style="margin-left: 40px;"> There's been some discussion in the blog world about using a Bayesian categorizer to enable a person to discriminate along various interest/non-interest axes. I took a run at this recently and, although my experiments haven't been wildly successful, I want to report them because I think the idea may have merit. [Full story: <a href="http://www.xml.com/pub/a/2003/11/19/udell.html">O'Reilly Network: Working with Bayesian Categorizers</a>] </blockquote> <div style="margin-left: 40px;">This month's O'Reilly Network column was a struggle because categorization itself is a struggle. I remain convinced that the automated classifiers that are doing such a good job beating back the tide of spam will also turn out to be more generally useful. But finding the right synergy between an automated assistant and a human overseer is a subtle and tricky thing. ... [<a href="http://weblog.infoworld.com/udell/">Jon's Radio</a>] </div> Great write-up by Jon of his experiments trying to hotwire Bayesian categorizers for auto-classifying blog posts. I think his conclusions demonstrate that there is value in the approach, certainly as Jon notes, for auto-classification of incoming items as in a news aggregator.  Of course this is exactly what we are doing with <a href="http://w4.evectors.it/">K-Collector</a> and I have mused for a while about whether we could use a Bayesian classifier to improve how we do that (at the moment we are using some pretty simple keyword analysis). I think one of the biggest obstactles is the training.  We have somewhere around 400 topics and many thousands of posts.  How hard would it be to train a classifier?  How long would it take?  Could you sell it? http://matt.blogs.it/entries/00001744.html Whose personality do you want today? Fri, 21 Nov 2003 17:47:08 +0000 <a href="http://www.decafbad.com/blog/">l.m.orchard</a> <a href="http://radiocomments.userland.com/comments?u=0107808&p=1243">commented</a> regarding the <a href="http://matt.blogs.it/2003/11/20.html#a1243">using Bayesian analysis on news</a>. In fact, as soon as I saw it I remembered, I had read his <a href="http://www.decafbad.com/blog/geek/bayes_agg_one.html">piece</a> already. It was probably his writing that triggered my initial interest in using a Bayesian classifier in <a href="http://w4.evectors.it/">K-Collector</a>. Re-reading that piece I got an interesting different angle since his approach was to blend a Bayesian classifier with his news aggregator to try and have it prioritize news he would find interesting and not to categorize it by topic. I think this is a much more scalable task, from a K-Collector perspective, than what <a href="http://weblog.infoworld.com/udell/2003/11/20.html#a851">Jon</a> is experimenting with. I think the efforts of training a system-wide recognizer to differentiate between topics would be too much for most users of the product to bear. Our product roadmap for K-Collector already includes allowing users to personalize the system. For example we think that people should be able to say which feeds they think are relevant on different topics. Notice that this is a much very granular relationship since it means that I can say "Matt Mower is a real expert on the topic sock puppets" but that this says nothing about how relevant I am on "dating." or any other topic. Indeed each user might rate the exact same sources differently over a wide range of topics. What might be interesting is if people could "share" and "subscribe to" preference maps. As a new user of the system you might not really know who is relevant on any particular topic. But imagine you worked with <a href="http://www.hyperorg.com/blogger/">David Weinberger</a>, <a href="http://www.dijest.com/aka/">Phil Wolff</a>, or <a href="http://weblog.siliconvalley.com/column/dangillmor/">Dan Gillmor</a>. If you knew them and trusted their judgement you could pick one of their preference maps as a starting point and immediately gain a usseful insight into the data as it is structured by topic. You might even switch between personalities to get more perspective! Thanks to l.m.'s piece I am now wondering also about whether a Bayesian classifier might be more use in helping users to establish their own preference maps about which content is most relevant to them. http://matt.blogs.it/entries/00001756.html Tar'n'feather spammers Sun, 10 Apr 2005 10:24:10 +0100 I've been looking for a good corpus to train a Bayesian classifier on (still looking). In my quest I've come across some interesting ideas and work. Today I came across one which I thought was pretty good indeed. Marty Lamb of <a href="http://www.martiansoftware.com/">Martian Software</a> (who seem to be a small Java software house) has published an idea to <a href="http://www.martiansoftware.com/articles/spammerpain.html">use statistics to cause spammers pain</a>. Philosophically the idea strikes me as quite similar to <a href="http://www.hashcash.org/">HashCash</a> which is a system for preventing denial of service attacks by forcing a client to do work before it can use resources. For normal users this work has little impact but when mounting a DoS attack it (theoretically) becomes prohibitive. Marty's idea is to bake a Bayesian filter into SMTP servers and have them analyze a message token-by-token as it arrives. With each token a spam probability is calculated and used to vary the the speed of the connection. As the server reaches the conclusion that a message is spam the connection slows to a crawl and it takes the spammer longer and longer to deliver the message. Marty describe it thus: <blockquote>If an incoming message looks like spam, the connection could be slowed dramatically, consuming the spammer's resources and wasting their time. This would transform the server into a sort of dynamic tarpit, in which the spamminess of the incoming message affects the viscosity of the tar. As the spam probability goes up, the socket speed goes down. If enough of these dynamic tarpits were in place (or just a handful were placed in the right places), the spammers' mail software would bog down, reducing the rate at which they can send messages, in turn reducing the fees they can charge their customers. If these tarpits were ubiquitous, they could completely change the economics of spam, creating a scarcity of bandwidth experienced only by spammers. Ouch. :) Of course, this doesn't eliminate the problem presented by careless or uninformed sysadmins who provide open relays; spammers going through open relays wouldn't feel a thing. But the sysadmins that provide those services certainly would. It isn't too far-fetched to imagine that they might start to notice the effect on their server when all of its outbound SMTP connections are slowed to a crawl.</blockquote> If this works then all we need to do is add feathers! http://matt.blogs.it/entries/00002486.html A Bishop and a Reverend walk into a bar Sun, 24 Apr 2005 21:59:29 +0100 Last weekend I ported the <a href="http://www.divmod.org/projects/reverend">Reverend</a> bayesian classifier from Python to Ruby (thanks <a href="http://homepages.unl.ac.uk/~alsburyj/">Jon</a> for the original pointer to Reverend). What I didn't realise was that <a href="http://tech.rufy.com/">Lucas Carlson</a> was, simultaneously, working a similar library... ah well, there's a good chance we'll roll my work into his more general classifier library. Anyway for what it's worth my port, <a href="http://rubyforge.org/projects/bishop/">Bishop</a>, is available for download and as a RubyGem. I anticipate at least one more release before deciding what to do with it. http://matt.blogs.it/entries/00002525.html Bayesian sillyness on a Sunday afternoon Sun, 24 Apr 2005 22:25:29 +0100 <a href="http://www.memescope.com/">MemeScope</a> is a project i've been working on for a little while now. The aim is to research ideas about RSS, tagging, and information management within a personal context. Working in <a href="http://www.ruby-lang.org/">Ruby</a> and utilizing the <a href="http://www.rubyonrails.com/">Rails</a> framework allowed me to make quick progress on the basic application however, lately, I've found it hard to make time with work and study (exams in May -- groan) pressing in on me. Today though I made time to do a little experiment using <a href="http://rubyforge.org/projects/bishop/">Bishop</a>. This has been done very quickly and crudely but I hope it will at least indicate a useful path to follow. Searching my bloglines subscription page I grabbed the first bunch of RSS2.0 feeds I could lay my hands on. They were (in the order in which I could remember whether these people used vanilla RSS2.0): <a href="http://paolo.evectors.it/">Paolo</a>, <a href="http://matt.blogs.it/">me</a>, <a href="http://radio.weblogs.com/0001011/">Scoble</a>, <a href="http://www.scripting.com/">Dave Winer</a>, <a href="http://www.hyperorg.com/blogger/">David Weinberger</a>, <a href="http://marc.blogs.it/">Marc Canter</a>, <a href="http://www.jackvinson.com/">Jack Vinson</a>, <a href="http://www.terryfrazier.com/weblog/">Terry</a>, and <a href="http://blog.mathemagenic.com/">Lilia</a>. I created a classifier and trained it with the feed of each person, using their name as a pool. Then I fed another couple of blogs through and looked at the guessing results. <a href="http://www.problogger.net/">Problogger</a>:<ul> <li>Jack = 0.468318829421744</li><li>Lilia = 0.00342010541710014</li><li>Marc = 0.282793689816727</li><li>Me = 0.393100714237438</li><li>Paolo = 0.346225200485668</li><li>Scoble = 0.253258334183292</li><li>Terry = 0.22771214945352</li><li>Weinberger = 0.0167048658202115</li><li>Winer = 0.289825786625603</li></ul> <a href="http://radio.weblogs.com/0110772/">Seb Paquet</a>:<ul><li>Jack = 0.576259503998576</li><li>Lilia = 0.235407686747673</li><li>Marc = 0.417682412318975</li><li>Me = 0.30854512908974</li><li>Paolo = 0.397301832715516</li><li>Scoble = 0.324509204588817</li><li>Terry = 0.291899188854501</li><li>Weinberger = 0.283141685408383</li><li>Winer = 0.108271483644474</li></ul> <a href="http://seblogging.cognitivearchitects.com/">Sebastian Fiedler</a>:<ul><li>Jack = 0.5</li><li>Lilia = 0.24819059477181</li><li>Marc = 0.5</li><li>Me = 0.234146370920656</li><li>Paolo = 0.326586572303767</li><li>Scoble = 0.317525809666623</li><li>Terry = 0.344058594866734</li><li>Weinberger = 0.208336788611639</li><li>Winer = 0.203113831786535</li></ul> The numbers here are probabilities indicating (as far as the classifier is concerned) the likely hood of the guessed feeds being the same as the sample feeds. There is probably far too little data here to determine anything useful however, given the people concerned, I do find something suggestive about the results. Enough to make me want to look harder. For calibration here are two other results. Feeding Paolo's feed back in for guessing:<ul> <li>Jack = 0.162912565311238</li><li>Lilia = 0.00398573502931349</li><li>Marc = 0.0498689666595031</li><li>Me = 0.00988167948078622</li><li>Paolo = 0.5</li><li>Scoble = 0.0404166297297532</li><li>Terry = 0.0308895504863602</li><li>Weinberger = 0.0169976173877616</li><li>Winer = 0.0083263139102045</li></ul> and for my own feed:<ul><li>Jack = 0.355604304977086</li><li>Lilia = 0.00691730756404335</li><li>Marc = 0.148014510140305</li> <li>Me = 0.5</li><li>Paolo = 0.0873206208996655</li><li>Scoble = 0.0863911457566834</li><li>Terry = 0.105989792649879</li><li>Weinberger = 0.0781196567440771</li><li>Winer = 0.00582495354516671</li></ul> Next time round I'm going to be more thorough, allow the use of RSS1.0 and Atom feeds, think about adjustments for text quantity and experiment with the use of the Robinson-Fischer algorithm for combining probabilities. http://matt.blogs.it/entries/00002528.html