Matt Mower: Tuning

Making blogs spider friendly

Tue, 05 Oct 2004 22:54:08 GMT

I've been playing with indexing blogs and, mindful of Anjo's comments, decided to try an experiment in making my blog spider-friendly. The original idea for this was Paolo's, we talked about it a while ago.

What I've done comes in 3 parts the first two of which are part of a Radio tool I will release shortly.

Write each weblog post into it's own separate RSS file. That's right, I now have about 1583 little XML feeds which each contain a single post. Here is an example feed.
Create a blogmap.opml file which maps each post from it's permalink to it's XML feed.
I add a <link> tag to my blog pages which says where to find the blogmap OPML file. This works the same way as the tag which says where to find your RSS feed.

The net result is that an application can now access the contents of my blog in a completely predictable way. From any page with the <link> tag (which, in theory, is every blog page) it should be able to locate the blogmap outline and from there it can find every post, or, using the permalink as a key, any individual post. Each post is encoded in it's own RSS file and can be read using any standard RSS2.0 parser.

Right now the tool has to be triggered manually and generates everything from scratch which is a complete waste of time (you don't want to upstream 1,000s of files every time). The next version should hook into Radio's publishing events to keep everything in sync quietly & automatically.

I would have thought similar implementations for MT, WordPress, TypePad and so on should be pretty trivial. Assuming, of course, that the idea makes sense in the first place...

STES weblog launched

Mon, 28 Jun 2004 23:42:27 GMT

Today we launched the STES weblog. In the coming weeks we hope this will be a great resource for learning about the event and the excellent group who will be leading our sessions.

RSS: It's a pretty diverse world out there

Fri, 11 Apr 2003 09:16:08 GMT

NetNewsWire/1.0.1fc1 (Mac OS X; http://ranchero.com/netnewswire/)

1,571

24.3%

13.9MB

6.79%

Syndirella/0.91pre

880

13.6%

6.8MB

3.31%

113

NewsGator/1.0 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.0.3705.209)

420

6.49%

1.2MB

<1%

Feedreader

406

6.27%

20.2MB

9.86%

MagpieRSS/0.3 (+http://magpierss.sf.net)

362

5.59%

2.3MB

1.13%

SharpReader/0.9.0.0 (.NET CLR 1.0.3705.288; WinNT 5.1.2600.0)

317

4.90%

3.6MB

1.76%

104

User Agent: Genome Machine (Powered by Waypoint); info@thinktank23.com

241

3.72%

38.2MB

18.6%

241

Java/1.4.1

211

3.26%

33.9MB

16.5%

188

Syndirella/0.9b

176

2.72%

1.4MB

<1%

NetNewsWire/1.0.3b1 (Mac OS X; Lite; http://ranchero.com/netnewswire/)

169

2.61%

2.2MB

1.06%

NetNewsWire/1.0 (Mac OS X; Pro; http://ranchero.com/netnewswire/)

159

2.46%

2.0MB

<1%

SharpReader/0.9.0.0 (.NET CLR 1.0.3705.288; WinNT 5.0.2195.0)

118

1.82%

716.2kB

<1%

Java/1.4.1_01

1.48%

8.6MB

4.20%

Java1.4.0_03

1.41%

6.6MB

3.20%

NewsGator/1.1 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.0.3705.209)

1.21%

625.4kB

<1%

NetNewsWire Lite/1.0.2b3 (Mac OS X)

1.21%

1.3MB

<1%

NewsGator/0.9 (http://www.newsgator.com; Microsoft Windows NT 5.1.2600.0; .NET CLR 1.0.3705.288)

1.19%

941.4kB

<1%

Frontier/8.0.5 (WinNT)

1.11%

1.8MB

<1%

Aggie 1.0 Release Candidate 5 - http://bitworking.org (Microsoft Windows NT 5.1.2600.0; .NET CLR 1.0.3705.288) http://bitworking.org/AggieReferrers.html

1.07%

1.8MB

<1%

Frontier/9.0 (WinNT)

1.02%

1018.2kB

<1%

Popdexter/1.0 (http://www.popdex.com/)

<1%

6.3MB

3.07%

AmphetaDesk/0.93.1 (MSWin32; http://www.disobey.com/amphetadesk/)

<1%

627.3kB

<1%

Python-urllib/1.15

<1%

4.8MB

2.32%

lwp-trivial/1.34

<1%

1.2MB

<1%

Syndic8/1.0 (http://www.syndic8.com/)

<1%

2.0MB

<1%

RssBandit 1.0d

<1%

2.2MB

1.09%

NetNewsWire Lite/1.0.2 (Mac OS X)

<1%

1.1MB

<1%

sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)

<1%

2.6MB

1.27%

Wildgrape NewsDesk

<1%

1.1MB

<1%

perl

<1%

2.4MB

1.18%

RPT-HTTPClient/0.3-3

<1%

1.4MB

<1%

Microsoft URL Control - 6.00.8862

<1%

3.2MB

1.56%

Java/1.4.1_02

<1%

4.8MB

2.36%

PHP/4.2.2

<1%

927.6kB

<1%

Lotus Notes

<1%

1.1MB

<1%

Gazette RSS Aggregator

<1%

959.1kB

<1%

Wget/1.7

<1%

326.1kB

<1%

Kevin http://websitealert.net/kevin/

<1%

2.9MB

1.42%

PHP/4.0.3pl1

<1%

727.5kB

<1%

asterias/2.0

<1%

1.2MB

<1%

Test#2

Wed, 22 Jan 2003 14:09:38 GMT

Test#2

Test

Wed, 22 Jan 2003 14:04:46 GMT

Test

Another test

Thu, 05 Dec 2002 16:58:25 GMT

Test post

Some good bits from the x-log

Wed, 27 Nov 2002 18:12:55 GMT

From x-log I found some useful stuff for advanced Radio users and Radio developers:

the rFrame macro for creating inline frames
the desktop event viewer displaying almost real-time the radio event log in your desktop website (very cool)

Tue, 26 Nov 2002 10:03:30 GMT

Test "^test2"

Test post

Tue, 12 Nov 2002 12:55:25 GMT

Continuing the testing. Do all the links work again now?

How to access the text in the weblog post editor form

Fri, 12 Jul 2002 22:45:34 GMT

This is a little tricker than it first seems because there are really two editors. The HTML textarea based editor and the WYSIWYG editor. What is even more confusing is that, if you are using the WYSIWYG editor, Radio defines a hidden input field with the same name as the textarea when it is used. This makes detecting which editor you are using a little more confusing!

Here is the code I ended up using

bundle { // Get the text of the blog posting

htmlText = htmlText + "if( thisForm.itemtext.tagName == \"INPUT" ) {" + cr;
htmlText = htmlText + "text = getEditorContents();" + cr;
htmlText = htmlText + "} else {" + cr;
htmlText = htmlText + "text = thisForm.itemtext.value;" + cr;
htmlText = htmlText + "}" + cr

}

TrackBack for Radio

Sat, 06 Jul 2002 00:19:26 GMT

David Watson has Radio talking to Movable Type's TrackBack feature. [Scripting News]

» Cool. I'll be looking to play with this ASAP.

Blogging by email

Sun, 30 Jun 2002 21:45:26 GMT

Radio wishlist > Post to email..

Dale Pike writes:

I want to be able to designate a category and have that post sent as an email message to a pre-determined address. This would allow me to further consolidate my communications and have a more streamlined "write once" approach to my messaging.

[a klog apart]

» I need exactly the same thing to keep legacy people in the loop. I'm trying to knock up something very quickly as a tool in Radio.

Basic features:

preferences per- subscriber email

filter by category & by liveTopic

immediately, hourly or daily feedings

send either complete post or permalink+title

I had originally thought about making it a program that subscribed to an RSS feed and emailed it out. However this seemed like a lot of work and a way of re-inventing my.userland. I'm trying to KISS!

TrackBack for Radio

Sun, 30 Jun 2002 19:54:33 GMT

I'm really excited by the TrackBack feature that has been implemented for MovableType. I'd really love to see something similar for Radio.

My first thoughts were to implement an RCS extension that uses XML-RPC calls. The idea is that whenever you post to your blog, an upstream callback extracts the links from the text and sends them to the RCS server. RCS would then keep a per-site record of all such pings. Another XML-RPC call would allow Radio to get information about all pings to any pages relating to your site. Working in the same ways as referrer information does now.

The one major disadvantage I see to this approach is that it isn't how MT's system works.

liveTopics progress

Mon, 24 Jun 2002 23:33:08 GMT

Once again I'm neglecting all other interests in the rush to make liveTopics a workable tool for Radio listeners. I'm learning what a struggle it is to debug problems on an unfamiliar OS over a 4,000 mile and 5 hours gap. Maybe I should just write better code, it'll be easier in the long run!

The good news is that two people now seem able to use liveTopics and hopefully by the end of play tomorrow there will be four. If we don't find any more serious bugs then I am on schedule to release version 1.0 at the end of the week.

Clever RSS

Sat, 22 Jun 2002 17:47:26 GMT

Now that postings can be annotated with topics to describe what they are about it would be interesting to be able to put those topics into the RSS feed somehow. This would allow clever aggregators to offer filtering tools to their users.

liveTopics

Fri, 21 Jun 2002 17:32:13 GMT

I haven't posted much the last couple of days because I've been working hard on my latest Radio tool. It's called liveTopics and you can see the results on this page. Some of the postings have a new Topics: section under them with links from various topics that the postings are related to.

Each link goes to a central Topic Table of Contents for my weblog. Where you can see all the postings related to a particular topic grouped together.

The tool also provides automatic generation of META tags with keywords and a "what i'm talking about" banner showing the topics associated with recent postings.

It should start being tested in the next couple of days and, if all goes well, I hope to release it next week.

More useful stuff

Tue, 18 Jun 2002 14:22:13 GMT

This one is from Lawrence and describes how to view the contents of the page table as well as some tricks for exploiting it. There are useful links in there too.

Description filter callbacks

Tue, 18 Jun 2002 14:04:29 GMT

Digging around for stuff written by Jake Savin I found something useful. I had been thinking that I didn't want my RSS feed to contain whopping great blocks of text (I haven't learned to blog in small pieces yet) and wondered how others were doing this.

UserLand introduced a callback that allows you to modify the description, to go in to the RSS feed, of each item being posted.

Regexes : another Radio gem

Tue, 18 Jun 2002 13:29:09 GMT

Yesterday as I was battling with the string verbs to extract some links from a page I found myself musing "Wouldn't it be great if Radio had regular expressions, that would make this easy..."

Well, you'll be pleased to learn that it does. Check out system.extensions.regex and in particular the README (you need to perform a simple step to initialize the package, I guess because it's an extension).

In my limited testing so far I've found them to be fast and easy to use.

Good stuff.

Post form callbacks

Mon, 17 Jun 2002 21:05:44 GMT

Jake Savin has released details of new callbacks for adding sections to the weblog post form. This allows you to request new pieces of data and process them when the posting is submitted.

A common request

Mon, 17 Jun 2002 15:28:18 GMT

I asked this at one time, I'm sure that most people probably do:

"How do I change the size of the editor in Radio?"

The answer should be to go to the prefs page, but unfortunately this one didn't make it. So here are the instructions:

You will need to open the native (i.e. non-web) Radio app.
Hit Ctrl+J and type in user.radio.prefs then hit Enter.
In the table look for the item browserBasedEditorSize which will probably be at the top and have a number in the next column.
Double click the number and change it to the number of rows you want.
Close the table & the radio app and reload the posting page.

Modding the WYSIWYG editor (for the adventurous)

Thu, 13 Jun 2002 16:39:57 GMT

For some time now I've wanted a way to insert characters (and one character in particular) into blog postings without resorting to charmap every time.

I've ammended the code for the wysiwyg editor to add this facility. It required finding it, then learning enough about the DHTML editor to figure out how to add text without blowing it up.

My modified wysiwyg outline (it lives in html.editor.wysiwyg) is here.

To install it:

open the table html.editor
rename the existing wysiwyg outline for safe keeping
load my wysiwyg from the OPML file
create a new outline in html.editor called 'wysiwyg'
paste the contents of the OPML file into the new outline
save and close the outline
load the weblog desktop site

you should have a new "character" drop down.

If anyone needs any help installing this just let me know. Also it only has 2 characters in it right now. I'll have a go at adding more when I know someone else needs them!

A simple but useful mod for the news view

Thu, 13 Jun 2002 11:11:17 GMT

I've modified the way the default news view works.

The way I always wanted this view to work was that it should open a new window whenever you click a link within the body of a story. Or if you click the globe icon for a site.

The recommended approach is a modification to %RadioHome%/www/system/pages/news.txt which adds as the first line #meta "<BASE TARGET='_blank'>".

This works, the links open in a new window, with the unfortunate side-effect that it also affects all the options from the editor menu and the delete button as well. This leads to a profusion of unwanted new windows that is quite irritating.

My solution is a modified version of the function radio.html.viewNewsItems() that addes the target='_blank' attribute to each link found within the body of a story. It also updates the links for the globe icon and the XML RSS feed as well.

Whilst I'm loath to modify an internal function since this will obviously not be much use if UserLand modify the function itself. I'm hoping that I can persuade UserLand to add this code as an option and make the patch unnecessary. (Note that no code is overwritten and you can switch back to the default functionality very easily).

To install get a copy of my function workspace.viewNewsItems() and install it in your own workspace table. Then open the news.txt file found in the folder /www/system/pages. Ammend the call to radio.html.viewNewsItems() to workspace.viewNewsItems() and you're done.

How to get the permalink for a posting

Sat, 08 Jun 2002 11:48:58 GMT

I'm writing a macro that processes the posting's in weblogData.posts and wanted to do something with the permalinks which are not encoded in the postings themselves. With thanks to Dave Winer I know have the official method:

radio.weblog.getUrlForPost()

This function returns true if it was able to generate the Url. I'm not clear on the reasons why this function might fail.

You call it by passing

adrPost - address of the table containing the post
adrUrl - address of a string that will be populated with the permalink
catname - name of the category the posting is in (default to an empty string)
adrdata - not sure what this is for, it seems to be configuration information in Radio

Comparing dates

Fri, 07 Jun 2002 23:46:40 GMT

I haven't found a function to compare dates in UserTalk yet so I'm doing something fairly simple like:

local( targetDate = clock.now() - ( numberOfDaysBack * 24 * 60 * 60 )
if( comparisonDate >= targetDate ) {
// do something
}

Which so far seems to work okay.

Digging in to publishing

Thu, 30 May 2002 15:24:50 GMT

Someone was asking on the Radio discussion group about how to publish a file to the cloud. Since I'm having problems with upstreaming I have a need to understand some of this as well and decided to take a quick peek under the hood.

If you want to understand how Radio publishing works, check out:

system.verbs.builtins.radio.menu.Publish

you can see the scripts that are invoked to publish & upstream various parts of your blog.

These seem to either delete entries in user.radio.settings.files, or use the function radio.weblog.publish() to publish individual files.

on add( s )

Tue, 28 May 2002 14:21:05 GMT

Okay there has to be a better way than this...

In my experience if you look inside most Radio macros that are generating HTML you will see something like:

local( htmlText = "", indentLevel = 0 )
on add( s )

htmlText = htmlText + string.filledString( "\t", indentLevel ) + s + cr

as a way of generating indented HTML on seperate lines.

I hate it.

There has to be a better way than this.

Community Builder

Tue, 28 May 2002 12:14:49 GMT

Going on with the idea of the Blog community builder (I'd love someone to suggest a cool name) here are some more thoughts:

This should be a tool that plugs in to Radio Userland like the Radio Community Server.
The tool allows people (including yourself) to register their blog.
Each registered blog is indexed on a regular basis (this might include indexing an incoming RSS feed rather than the site itself)
Each index creates a set of likely keywords.
The blog owner is notified that their blog has been indexed (email?). They can then sign in and modify the keywords that have been suggested and then rank them in order of importance.
Some weighting algorithm is then applied to the keywords for the blog to try and match it up with likely candidates.
The candidate list is filtered against a list of already generated candidates as well as those to which the blog already links.
This candidate list is then provided back to the blog owner. Options: email, web page or RSS feed.

The reason I think this should be a Radio tool is that it can then be run by many people. Some attempt should be made to allow these to tools to federate and share index information. The index will need good algorithms for matching blogs up.

It seems to me that there is also a temporal issue here. If my blog is all about, for example, Scuba diving and then I happen to post one day about my Dell Inspiron laptop what do I want to happen? I suppose this may be solved by repeated keyword filtering (i.e. I don't allow 'dell laptop' to become attached to my blog) but I guess I would rather the system have some clever way of evolving its representation of your interests.

Indeed, is there some way around having to know about and rank keywords in the first place?

Creating communities from thin air #2

Tue, 28 May 2002 10:08:18 GMT

I've had some interesting comments about my previous post on creating communities. It seems like it is, at least, not a stinker!

One simple thing that came up was that we should be able to attach meta-data to our postings. I had already been wondering about the idea of adding a keywords field to the posting box backed up with some simple UserTalk to automatically grok the new posting for suitable keywords for you. I'm no RSS expert but I'm sure somebody said that new tags could be added without breaking existing applications. This means we could introduce meta-data and let the applications catch up.

Another idea that came up was that this should not be a centralized service, that it should be democratized. I think it's a good point, but wonder how it can be achieved? The value of the service would be in making good recommendations and so would lie in the index. Distributed systems that don't hold the full index won't be able to make many recommendations (unless we come up with some kind of brokering service, or a distributed search or something like that). I'd love to hear suggestions on how this problem could be solved.

Since the basic idea doesn't sound like a dud I'm going to start working on it. I'd love to hear from anyone who would like to collaborate with me.

Playing with the aggregator / transBlogging

Fri, 24 May 2002 19:17:50 GMT

The Aggregator is the piece of Radio that downloads news items from RSS feeds on a regular basis. I have an idea for something I'm calling the 'transBlogger' which will automatically syndicate content from the Aggregator.

Here's the idea, silly as it may be:

There may be blogs/bloggers that you respect and, whilst you may not agree 100% with every post, you think they are worthwhile and should be widely disseminated. You are probably subscribed to their RSS feed but may not want to have to approve/comment on every item.

The transBlogger will allow you to choose to automatically syndicate their content to your own blog. It will take the latest story or stories (you choose how many) and put them in a special area on your weblog home page. This will be controlled by a macro so you decide where and how they should look (within certain boundaries).

Important: Each story will be clearly identified by source and will be linked back to the source. You aren't appropriating their content, just pushing it to a wider audience.

Of course nothing stops you from also posting items from news with your own comments. Indeed you'll still want to do this as the transBlogged content will change quite quickly as new stories pile up in the aggregator.

I very, very loosely based this idea on Tedd Nelson's concept of transpublishing. In fact probably only in so much as I was thinking about transpublishing the other day when I also thought of this.

But there ya go.