There is constantly a lot of chatter regarding the business model of twitter and how, as a service, it is going to make money.  Along with this, there are continual debates regarding the growth, decline and overall engagement of users who are participating in the micro-blogging service.  Regardless, there is no doubt that there are few other vehicles that can accomplish the viral buzz and reach potential than twitter.

I’m a firm believer that people, and history, repeat themselves often over centuries, decades, years, months or even every few minutes.  As as student of history, I believe it is important to follow what people are talking about today but more importantly it is beneficial to understand the conversations they had in the past.  This helps us understand trends and consumer interactions, track market conditions and see how they impacted online conversations, view seasonal consumer opinions and better understand how past events, campaigns and promotions impacted user discussions.  As marketers, this information can be invaluable in planning the next big idea and forecasting both consumer and competitive response.

There are plenty of web-based tools (search.twitter.com, hootsuite, tweetdeck, twiogle.com to name a few) which provide a good interface for searching current trends but all seem to be lacking when searching phrases beyond the last ten days especially if your search is based upon a phrase and not a username or hashtag.  Let’s not forget, the value of a conversation is not in hashtags but the context and scope of the discussion.  Given the immense volume of data stored within the twitter database’s I’m not surprised this problem exists.  Sure, the tweets are only 140’ish characters but creating robust indexes on hashtags, usernames and full text lookups and then combine this with date ranges can be a data mining nightmare for even the most seasoned dba’s.  I’m certain at some point twitter will find a way to better expose this information for marketers to use.  Could this be a subscription service and revenue model for them?  Perhaps.

Over the past several weeks I have been working on a side project to design and build a better historical conversation archiving mechanism. During this process I’ve identified that we certainly could not keep track of all the conversations online as we would have the same data mining challenge that I mentioned before.  We can however identify topics and trends we want to track and archive those over time.  From this data, we can then model our own searches based upon historical conversational information captured along the way.

Have you had the same problem?  Do you find yourself needing the same information?  I’m looking for a few individuals or groups who want to participate in the product planning and alpha/beta testing of this application.  If you’re interested contact me or send me a direct message at @JeremySchell.  As this project evolves I’ll post the details, and possibly (maybe) source code for everyone to view.