December | 2005 | Gordon Luk

Hi getluky.net readers!

As you might know, this blog occasionally plays host to news about Freetag, the open source folksonomy module that I maintain. Since the beginning, i’ve had grand plans of federated Freetag instances talking with one another to make some interesting connections and give developers some cool tools for connecting applications.

I’ve got some ideas, but I would love to hear from anyone what other concepts are out there, and get some comments about this plan so far.

The basic idea

The basic idea behind federating Freetag installations would be to allow a single query to a master system (probably hosted by me or something) to spread queries out to all the separately-maintained various public Freetag instances out there.

Essentially, you’d search the master system for ‘music’. All databases that use Freetag would then be queried for any objects tagged with ‘music’. The query results would be a mixture of all the stuff from various databases tagged with ‘music’, such as music events on Upcoming.org, spots with live music for eatlunch.at, etc. The system would probably be kinda slow (hey, sorry, I only run one personal server right now. :p), but it would be cool!

The interesting questions are how to implement such a system. These are some of my ideas:

Simple, extensible REST gateway

I could create a fairly flat REST gateway PHP page that would allow for some common queries to be handled by each Freetag instance. Since most Freetag queries produce a list of integer id’s that app developers use internally to access their databases, i’d also provide a skeleton function that a developer would essentially add a SQL statement to, which would collect a recordset of their internal data. The gateway page/class would then run the query locally, and then automatically construct a fairly common XML structure out of the results which could then be parsed by the master.

Creating a Freetag App Registry

I will probably do this in any case, but it poses some interesting federation questions. People email me all the time letting me know that their application uses Freetag. It would be cool to let developers add their app & URL to a list of Freetag-powered applications. I’d be happy to display a logo and link to each one at the bottom of my Freetag page after review for appropriateness.

Anyway, back to federation. This would allow for only trusted, tested Freetag installations to be queried by the master. It would help ensure quality of results, too. For example, if one system just gets too slow and makes the rest of the results tricky, then we could disable that host until the problems go away. I’d certainly work hard to ensure that all responses are adequately parsed for security, but any malicious hosts could be shut down from the master as well.

I know that this creates some problems with fragility (a single point of failure being the master), I’m not sure how a p2p system would work.

The open-source question

I’m not convinced that the code for the master would be something I want to open source like Freetag. I know that hosting the master is more of a service than anything, and i’d probably want to run some contextual ads in the sidebar to cover costs and, in a perfect world, be a way for me to profit from my work on Freetag.

I’m mostly happy with getting more and more sites on the tagging bandwagon. And believe me, there have been far more sites using it than i’ve blogged about, and i’m always happy to hear about the new stuff sprouting up.

Interface

It would be cool to provide an AJAXy interface to this master db, similar to Adactio Elsewhere. I’m not certain that’s the best way to go about it, either, as it’s a remarkable amount of querying over federated systems.

I will definitely be implementing some level of caching, and i’m also liking the idea of displaying and allowing access cached queries via a search interface that essentially uses the GUIDs returned by the federated systems to ensure that updates replace the original records of objects in other databases. It would essentially provide non-slow access to the wealth of interesting data that is out there in Freetag installations.

Terms of Service

Of course, providing such a federated system probably requires multiple parties to be in agreement not only about the technical communications protocol, but also about the ownership and redisplay of content via a master interface.

If it’s not OK with an app’s users to redisplay their content on a master Freetag query interface (or via API), then that app should not be a part of the federated system. Period. Remember, this would probably start as just something cool, but I expect it to consume a LOT of resources and I will almost certainly run ads.

You could look at it as a specialized gateway for crawling a database-backed site, which can also pull interesting metadata that you control for display to end users. Think of it as a tag-powered search engine, that can look directly into your systems. A federated Freetag Base, if you will. If you don’t mind the fact that Google redisplays your website and shows ads on your content, then you probably won’t mind this kind of usage as long as the master properly caches queries and doesn’t run wild. Besides, you can run ads on your own site, to which i’m sure the master interface would provide lots of traffic. Oh, that also would mean you’d have to provide me with a callback URL to which I could provide the object ID of an object that a master interface user could click on to get back to your original site. I’m open to hearing your feelings about this as app developers, as I would want as many sites to participate as possible, but it’s not something i’d want to do for free.

The other party to consider are the end users of the master interface. These users would most likely be addressed with a standard TOS, though.

Conclusion

So, those are my current thoughts. A registration-required network of participating Freetag installations providing a master query engine with a standard interface, defining the format their own data is returned in. I’m sure there are plenty of issues here, so please leave some comments so we can discuss it!

By request, here’s a quick function I whipped up to generate some basic tag cloud HTML from the output of the silly_list function:

[php]
    function get_tag_cloud_html($freetag, $num_tags = 100, 
  $min_font_size = 10, $max_font_size = 20, 
  $font_units = 'px', $span_class = 'cloud_tag', 
  $tag_page_url = '/tag/') {        
        $tag_list = $freetag->silly_list($num_tags);
        // Get the maximum qty of tagged objects in the set
        $max_qty = max(array_values($tag_list));
        // Get the min qty of tagged objects in the set
        $min_qty = min(array_values($tag_list));
        // For every additional tagged object from min to max, we add
        // $step to the font size.
        $step = ($max_font_size - $min_font_size)/($max_qty - $min_qty);

        // Since the original tag_list is alphabetically ordered,
        // we can now create the tag cloud by just putting a span
        // on each element, multiplying the diff between min and qty
        // by $step.
        $cloud_html = '';
        $cloud_spans = array();
        foreach ($tag_list as $tag => $qty) { 
            $size = $min_font_size + ($qty - $min_qty) * $step;
            $cloud_span[] = '' . 
    htmlspecialchars(stripslashes($tag)) . '';

        }       
        $cloud_html = join("\n ", $cloud_span);

        return $cloud_html;

    }

Feel free to tweak as you please. I wrote the code and tested by adding a tag cloud to eatlunch.at, so I can guarantee that it works. $freetag is supposed to be an initialized freetag object.

As a side note, eatlunch.at is horribly unmaintained right now. I’ve got to rewrite it soon.

WordPress is a TOTAL NIGHTMARE for posting code.

Gordon Luk

Monthly Archives: December 2005

Freetag v0.240 – bugfixes & enhancements

Request for ideas on Freetag federation

By Request: Tag Clouds with freetag::silly_list