Why Your App Needs Automated Data Export

If you have an app of some kind in which people store data, then everyone probably wants an API. Developers somewhere are clamoring for you to open a web service (SOAP, REST, protocol du jour, whatever) for it so they can do their own stuff.

I hope you have one which you planned from the start and built in from the ground up. Furthermore, I hope this API is the API and that your own web interface runs off of it too.

No? Okay, then I hope you’re planning this, although I warn you that wedging it in after the app has been built is probably going to suck, cost more, and take longer than you expect.

So, in the meantime, do this instead –

Build a solid data export API that works unattended. Build something whereby someone can call a URL securely with their credentials as simple querystring arguments and get back a data dump as a response. Just a zip file of XML documents or some other format representing a complete snapshot of their data in your system at that moment.

Why? Because then they can get their data out on a scheduled basis, and mash it up on their own infrastructure. This type of API is far easier to build than a “real” read/write API. It’s read-only, and needs no interface – just a URL that accepts credentials and responds with the data.

Why is this useful? I can give you two examples –

The first is Episerver’s licensing center. Episerver stores all our client’s license information on their servers, to which they have a web interface. This is fine, but we want to do some reporting they don’t cover.

If I could get an XML dump nightly of this data, I could write a script to grab it, unzip it, import it into SQL Server, run some SQL statements, then send out alert emails if the results warranted them (upcoming license expirations, for example). Then I’d throw all the data away and do it again the next night.

(Inefficient? Sure, but who cares? I can write a script to do this overnight, unattended, in less than a minute.)

The second example is Insightly, which is a CRM system based on Google Apps. We love it, but – again – we want some reporting they don’t offer. They keep promising a full-blown API (whoops – see update below), but if they would just give me an automated data export API, this would let us get started with integrations that don’t require read-write access.

For instance, on a client record in Insightly, we’d like to have a field for the person at Blend who works with that client. When a inbound call comes into the phone switch, we’d like to do a lookup to see if we can match the calling number to a client, figure out who the client probably wants to talk to, and automatically transfer the call to that person.

I do not need write access to do this. Furthermore, I do not need real-time access. The lookup table of clients, phone numbers, and account managers can be 24 hours old (a week old, even) and it would still be 99.9% accurate. I could do this with a simple, scheduled dump of my data. I’d write a quick import to a MySQL table which I’d clear and reload every night.

I absolutely believe in well-planned read/write APIs that embrace protocols and give someone complete programmatic control of their data. But those are complicated to do well, require lots of planning, and need to evolve with your application’s development.

Simple data export is so much easier, and it gets people developing against their data in your system sooner.

Do this early, then watch what people build. They’ll surprise you, and they’ll give you some fantastic insights on what they want to do with the data you manage. The eventual web service API you build will be better for it.