HamsterSoup - Recipes for Data Geeks

Freebase Hack Day – Open Government session by Greg Elin

Will Moffat — Tue, 11 Nov 2008 00:31:01 +0000

Greg Elin, representing the Sunlight Foundation gave a talk at the recent Freebase Hack Day on “Government data sources, transparency opportunities, data catalogs and the Sunlight Foundation”.

Personally I think that Freebase would be an great platform for working on open government projects. Greg seemed very positive about Freebase too. Sunlight’s database IDs for US politicians are currently being loaded into Freebase as I type, so hopefully we’ll be able to make some nice political demos soon.

Greg’s talk was a very informative summary of who’s who in open data in the US, here are my raw notes:

The Sunlight Foundation give away ~$3m a year to fund tech projects in open government. They have a good summary of the government data out there.

eGovernment is generally supported in US, improvements rolling along, 1st generation IT initiatives, transparency is a given now, (especially thanks to the financial crisis) – but government big and slow and needs to be encouraged in the right direction, hence the Open Government Data Principles

Every big org’s website should have a “data catalog” page which is like an “about” page.
theinfo.org by Aaron Swartz – for people with large datasets (lots of government stuff) political parsers – critical mass of group of people parsing political data, NY times, FEC, – code & techniques for parsing

watchdog.net data about representatives, cool visualizations + data feeds, big data dumps

public.resource.org – lots of bulk data

govtrack.us – xml dumps, scraps GEO, THOMAS –> XML

programmable web.com/government – tracking new government apis and mashups

everyblock.com – super local what’s new feeds

mysociety.org – high impact UK org, tracks foi requests, fixmystreet.com – tracks repsones very well

Freebase search command for Ubiquity

Will Moffat — Wed, 27 Aug 2008 14:46:02 +0000

I just wrote a Freebase search command for Ubiquity:

If you don’t know what Ubiquity is then watch the video.

Ubiquity is going to be awesome for Firefox power-users. One of the best things about Firefox is the extension eco-system and Ubiquity is going to make writing extensions much easier.

P.S. Ubiquity is only on v0.1 and executes 3rd-party code on your machine. You probably shouldn’t install it in your main Firefox profile just yet.

Freebase London meeting

Will Moffat — Mon, 18 Aug 2008 14:10:56 +0000

I’ll be at the Freebase London meetup on Wednesday night. If you’re interested in mashing-up open data, server-side javascript or just curious about Freebase then do come along and say hi.

One of the topics of conversation is sure to be the awesome prototype Freebase interface that David created, Freebase Parallax. Read the release notes and watch the video, it’s 8 minutes but it really gives you an idea of the power of Freebase.

Hang Firefox 3 on Mac with Firebug

Will Moffat — Fri, 04 Jul 2008 12:27:20 +0000

This 2-line web page will hang Firefox 3 with Firebug on a Mac:

(With Firebug scripting enabled)

This completely hangs the Firefox user-interface, so you can’t access your other tabs. The only thing to do is a Force-Quit. I reported the bug but unfortunately I just don’t have the time and energy to learn how to fix it myself. Hoping somebody out there might have some useful insights.

MQL: Finding the Wikipedia page for a Freebase Topic

Will Moffat — Tue, 27 Nov 2007 14:25:37 +0000

Freebase contains many articles mined from the Wikipedia. For example, this page about Edinburgh.

You might want to show the Wikipedia page in your own web application. Given the Freebase topic ID, in this case /en/edinburgh

This MQL query will return the Wikipedia ‘curid’

[{ "id" : "/en/edinburgh", "key" : [{ "namespace" : "/wikipedia/en_id", "value" : null }] }]

Run in Query Editor

Result:
{ "code":"/api/status/ok", "result":[{ "id":"/en/edinburgh", "key":[{ "namespace":"/wikipedia/en_id", "value":"9602" }] }] }

The Wikipedia URL is then:
http://en.wikipedia.org/wiki/index.html?curid=9602

Update: You can find the Wikipedia article name + all the redirect names with this query but as far as I know there’s no way to distinguish between redirects and the current article name.

Tagging demo with Freebase Suggest

Will Moffat — Thu, 08 Nov 2007 12:11:24 +0000

I whipped up a quick example of tagging with Freebase Suggest.

This is just the client-side user interface. Integration with an existing tagging system is left as an exercise for the reader

Freebase Suggest demos

Will Moffat — Sun, 14 Oct 2007 10:59:08 +0000

As an excuse to avoid packing my suitcase (an activity which I detest about as much clothes shopping) I wrote a few examples for Freebase Suggest.

The live demos are on the Freebase Suggest project page.

(Geek note: The 3rd demo includes a first attempt at a Freebase MiniTopic plugin for jQuery. This is a really easy way to generate an infobox or summary of a Freebase topic for your own page)

Freebase apps without cookies

Will Moffat — Thu, 09 Aug 2007 20:22:36 +0000

Freebase just announced that

Within a week or so we will be removing the registration requirement in order to view Freebase.

Soon all my demo apps will work without requiring a Freebase invite. Cool!

Avoiding timeouts – Using MQL cursors with MJT

Will Moffat — Sat, 04 Aug 2007 15:56:19 +0000

If you start to develop Freebase applications you then at some point you will inevitably see the error message “Query timeout”

For example, if you wanted to get a list of Musical Artists in Freebase you might try this query

{
“query”:[{
"id":null,
"limit":0,
"name":null,
"type":"/music/artist"
}]
}

But as there are almost 1/3 million artists in the database this query will timeout. That’s just as well, otherwise you would be loading a 30MB JavaScript file!

That was an extreme example, but as more users register with Freebase, my Map of Freebase Users occasionally timeouts. (It’s actually the MJT library which imposes the deadline here, not necessary the database which timeouts). Even if your application isn’t timing out, you find that your browser becomes unresponsive as it loads large JavaScript files from the database.

The solution is to use MQL cursors

I’ve created an example using MQL cursors in MJT which demonstrates a number of concepts:

Using MQL cursors
Calling MJT templates with arguments
Incremental queries (the page updates without reloading)

And we’re back

Will Moffat — Thu, 26 Jul 2007 07:00:51 +0000

Our short caving expedition is already finished. The Canto system is now almost 80km long!