Curiouser and Curiouser!

Me and my research

I read Brad Neubergs post Creating a Personal Research Agenda yesterday and it lead me to actually reading the paper of Richard Hammings talk You and Your Research at last. The thrust of Hammings talk was about "doing great work" and why so many people who have everything going for them don't seem to amount to anything great (although many amount to something good).

Hamming sums things up this way:

I claim that some of the reasons why so many people who have greatness within their grasp don't succeed are: they don't work on important problems, they don't become emotionally involved, they don't try and change what is difficult to some other situation which is easily done but still important, and they keep giving themselves alibis when they dont. They keep saying it is a matter of luck. I've told you how easy it is; furthermore I've told you how to reform. Therefore, go forth and become great scientists!

Now I'm not a research scientist. I'm not paid by anyone to do research. But this smacks a little of an alibi. I think in this blog back in 2002-3 I was getting close to at least defining some important problems to work on. Then I went off the boil.

Reading and re-reading Hammings paper has refocused me on getting involved again. At the moment my interests are still very eclectic but there are themes:

I am reading about factor analysis. What interests me about factor analysis is that it is a tool that allows you to discover structure. I am also interested in tools like clustering which also allow you to discover structure but, in many cases, they seem to require guidance. For example when using a clustering algorithmn like K-means you decide how many clusters you want and, often, the order in which you feed things to be clustered affects the outcome. Factor analysis is a statistical technique closely related to pychology. I'm not sure how widely used it is outside psychology. It's an area I would like to explore.

The structure of information and of relations between information has always been of interest to me. K-Collector was, at heart, a way of exploring the structure of related information through the evolution of a taxonomy. It was a primitive tool by the standards of what will come but I think it was an interesting one. I'm still very interested in tagging and tagging systems. I'm especially interested in faceted classification because it exposes the underlying structure of the taxonomy in what I think are useful ways.

I'm interested in what we we do with information, how we structure it and how we make use of it, how we filter it, combine it, and transform it. Information has never been so malleable as it is right now and this trend will only increase. I think links are fascinating and a good starting point for this problem. I am also interested in how we see information, how we visualize it.

Now this is just a set of interests. What it is not is a list of important problems, great problems. I think this is where I went wrong, I lost sight of the problems. It's hard to be emotionally committed to an interest, it's too nebulous. But a really thorny problem that holds out great reward is something you can commit yourself to and, in many cases, permits no other response.

But I am also aware that it's very hard to focus like this. Hamming himself did not try to work 100% on great problems but set aside Friday afternoons for it. He figured that 10% of his time, compounded over a long period, would be significant. I wonder if this is behind Googles 20% "personal time" practice although I am not clear that people are encouraged to work on "great" problems with that time.

I need to reformulate my interests into a set of problems and work out whether solving them amounts to a hill of beans.

Tunnelling BitTorrent over SSH

I'm making a note in case I hit this particular wall that you can tunnel BitTorrent over SSH. This came up when I read a post about Comcast breaking BitTorrent uising a technology called SandVine.

The usual ISP argument goes like this "If everyone uses the bandwidth they've paid for our network will collapse." This argument is horse shit. What the ISP is saying is that they are massively overselling their bandwidth and if people used what they thought they were paying for the ISP couldn't cope. It's like when I turn up to the airport to find that my flight is "overbooked." How do you sell more tickets than you have seats? Willfully, that's how.

I would counter that ISP's should have their feet put to the fire and sell what they can provide at prices that are profitable. That gets rid of "fair usage" at a stroke. It's not about some arbitrary measure of "fairness" but about what I have paid for and can legally use. Of course the ISP's and Telcos will fight this bitterly because it would put them on a level playing field where they could no longer operate a cartel and would actually have to compete leading to a reduction in price and improvement in service.

Our problem as consumers is that telcos & ISP's have lobbied so successfully to corrupt the law and strange all methods of competition that as consumers we are forced to put up with them if we want an IP dialtone.

To the trough of disillusionment we go!

I read a post on Paul Walks blog some time back: Tagging: Are we in the Trough of Disillusionment? where Paul is enquiring as to the state of tagging.

I started tagging on this blog back in June of 2002 and, with Evectors, I co-created K-Collector which was all about tagging. I think I know a little bit about tagging and the value of tagging. From what I can see we are nowhere near the "Slope of Enlightenment".

I have been surprised, disappointed, and excited that, despite the widespread adoption of tagging across many applications, the state of the art in tagging seems firmly wedged in 2003. Surprised because there seemed, despite the expectations of many that nobody would tag things, to be a momentum building in the use of tagging. Dissappointed because I expected to be using applications that really used tagging to do some interesting things. Excited because it means the field is still open.

I think the uses of tagging definitely forms part of my personal research agenda. Although I can't claim to have done anything innovative in the field since 2004 I have not stopped thinking about it and have ideas on the drawing board that will hopefully see the light in 2008.

Q: How can we develop tagging to fundamentally change how we think about and explore ideas?

Tagging in 2007 seems to have advanced no further than a means by which one or more users of a site (or application) can group content around a loose framework of concepts. If you are lucky those concepts are may express relationships but often they do not. In K-Collector we expanded tagging to include relationships as well as a prototype faceted classification along the lines of Who, What, Where.

What new developments in the use of tagging and classification systems will change the way people manage, transform, filter, and use the kinds of information they are already working with? What sort of applications will benefit most from advances in tagging?