Creating persistent APIs for any website: The Secret Sauce is Dapper
I written about creating an API for any website before, but those methods were fraught with danger if the underlying site changed its structure. I’ve known about dapper for quite some time but I didn’t realize that dapps are resilient against underlying site changes until now.
The “Dapp Factory,” a Rhino-based server application and a web front-end that deals with just about any site by proxying your requests and modeling the DOM on the proxy, then recording your actions for later replay. But, their secret sauce is a super-cool algorithm that figures out the structure of pages in such a way that your API can withstand changes to the target site, making your feed resilient to all but massive site overhauls. You then simply consume an XML or JSON feed, or use a simple API to dynamically construct paramaterized feeds.
Pivotal Blabs: Screen Scrape no more…Seriously! by Parker Thompson emphasis mine
Go find an existing dapp or create a dapp for any site now!.
In the spirit of mocking my friend Parker… I mean full disclosure, I’m employed by Pivotal Labs and am friends with Jon of Dapper.
Hey Wes!
So yeah, Dapper is pretty resilient to underlying site changes. Naturally, if a site owner completely overhauls the structure of the page, your Dapp will stop working. To combat that, we currently monitor Dapps and alert the creator via email when the Dapp’s quality begins to deteriorate. In the coming months, we’ll be completing a major project which automatically corrects Dapps as they deteriorate in a pro-active manner. This technology will actually also be used to create Dapps automatically for some sites, and it employs some really cool methods.
I’ll catch you up over a beer next time you’re in town.
Jon
Hi Wes,
I’ve tried to use Dapper a lot to create feeds from websites and, each time, despite great promise, the Dapp hasn’t come out working properly for me. However, last time this happened (last week) I received good support from the Dapper team and they helped me out creating this Dapp: http://www.dapper.net/dapp-howto-use.php?dappName=EarthCalendarWorldHolidays
It takes two variables, a month and a day, and returns the national holidays around the world for that date.
I think that when Dapper manages to solve some of the functional problems with its Dapp Factory, it is poised to really take off. I haven’t experienced a good competitive product yet. Any tips?
J.