Making blogs spider friendly
Tue Oct 05 23:54:08 BST 2004 Permalink
I've been playing with indexing blogs and, mindful of Anjo's comments, decided to try an experiment in making my blog spider-friendly. The original idea for this was Paolo's, we talked about it a while ago.
What I've done comes in 3 parts the first two of which are part of a Radio tool I will release shortly.
Right now the tool has to be triggered manually and generates everything from scratch which is a complete waste of time (you don't want to upstream 1,000s of files every time). The next version should hook into Radio's publishing events to keep everything in sync quietly & automatically.
I would have thought similar implementations for MT, WordPress, TypePad and so on should be pretty trivial. Assuming, of course, that the idea makes sense in the first place...
What I've done comes in 3 parts the first two of which are part of a Radio tool I will release shortly.
- Write each weblog post into it's own separate RSS file. That's right, I now have about 1583 little XML feeds which each contain a single post. Here is an example feed.
- Create a blogmap.opml file which maps each post from it's permalink to it's XML feed.
- I add a
tag to my blog pages which says where to find the blogmap OPML
file. This works the same way as the tag which says where to find
your RSS feed.
Right now the tool has to be triggered manually and generates everything from scratch which is a complete waste of time (you don't want to upstream 1,000s of files every time). The next version should hook into Radio's publishing events to keep everything in sync quietly & automatically.
I would have thought similar implementations for MT, WordPress, TypePad and so on should be pretty trivial. Assuming, of course, that the idea makes sense in the first place...

