location based blog: Discovery Is A Problem That Will Always Need To Be Solved

This past week I spent a fair amount of time in New York, meeting with smart folks who collectively have been responsible for funding and/or starting companies as varied as DoubleClick, Twitter, Foursquare, Tumblr, Federated Media (my team), and scores of others. I also met with some very smart execs at American Express, a company that has a history of innovation, in particular as it relates to working with startups in the Internet space.

I love talking with these folks, because while we might have business to discuss, we usually spend most of our time riffing about themes and ideas in our shared industry. By the time I reached Tumblr, a notion around "discovery" was crystallizing. It's been rattling around my head for some time, so indulge me an effort to Think It Out Loud, if you would.

Since its inception, the Web has presented us with a discovery problem. How do we find something we wish to pay attention to (or connect with)? In the beginning this problem applied to just Web sites - "How do I find a site worth my time?" But as the Web has evolved, the problem keeps emerging again - first with discrete pieces of content - "How do I find the answer to a question about...." - and then with people: "How do I find a particular person on the Web?" And now we've started to combine all of these categories of discovery: "How do I find someone to follow who has smart things to say about my industry?" In short, over time, the problem has not gotten better, it's gotten far more complicated. If all search had to do was categorize Web content, I'd wager it'd be close to solved by now.

But I'm getting ahead of myself.

Our first solution to the Web's initial discovery problem was to curate websites into directories, with Yahoo being the most successful of the bunch. Yahoo became a crucial driver of the Web's first economic model: banner ads. It not only owned the largest share of banner sales, but it drove traffic to the lion's share of second-party sites who also sold banner ads.

But directories have clumsy interfaces, and they didn't scale to the overwhelming growth in the number of websites. There were too many sites to catalog, and it was hard to determine relative rank of one site to another, in particular in context of what any one individual might find relevant (this is notable - because where directories broke down was essentially around their inflexibility to deal with individual's specific discovery needs. Directories failed at personalization, and because they were human-created, they failed to scale. Ironically, the first human-created discovery product failed to feel...human).

Thus, while Yahoo remains to this day a major Internet company, its failure to keep up with the Internet's discovery problem left an opening for a new startup, one that solved discovery for the Web in a new way. That company, of course, was Google. By the end of the 1990s, five years into the commercial Web, discovery was a mess. One major reason was that what we wanted to discover was shifting - from sites we might check out to content that addressed our specific needs.

Google exploited the human-created link as its cat-herding signal. While one might argue around the edges, what Google did was bring the Web's content to heel. Instead of using the site as the discrete unit of discovery, it used the page - a specific unit of content. (Its core algorithm, after all, was called PageRank - yes, named after co-founder Larry Page, but the entendre stuck because it was apt).

Google search not only revolutionized discovery, it created an entire ecosystem of economic value, one that continues to be the Web's most powerful (at least for now). As with the Yahoo era, Google became not only the Web's largest seller of advertising, it also became the largest referrer of traffic to other sites that sold advertising. Google proved the thesis that if you find a strong signal (the link), and curate it at scale (the search engine), you can become the most important company in the Internet economy. With both, of course, the true currency was human attention.

But once again, what we want to pay attention to is changing. Sure, we still want to find good sites (Yahoo's original differentiation), and we want to find just the right content (Google's original differentiation). But now we also want to find out "What's Happening" and "Who's Doing What", as well as "Who Might I Connect With" in any number of ways.*

All of these questions are essentially human in nature, and that means the Web has pivoted, as many have pointed out, from a site- and content-specific axis to a people-specific axis. Google's great question is whether it can pivot with the Web - hence all the industry speculation about Google's social strategy, its sharing of data with Facebook (or not), and its ability to integrate social signal into its essentially HTML-driven search engine.

While this drama plays out, the Web once again is becoming a mess when it comes to discovery, and once again new startups have sprung up, each providing new approaches to curate signal from the ever-increasing noise. They are, in order of founding, Facebook, Twitter, and Tumblr, and oddly enough, while each initially addressed an important discovery problem, they also in turn created a new one, in the process opening up yet another opportunity - one that subsequent (or previous) companies may well take advantage of.

Let me try to explain, starting with Facebook. When Facebook started, it was a revelation for most - a new way to discover not only what mattered on the Web, but a way to connect with your friends and family, as well as discover new people you might find interesting or worthy of "friending." Much as Google helped the Web pivot from sites to content, Facebook became the axis for the Web's pivot to people. The "social graph" became an important curator of our overall Web experience, and once again, a company embarked on the process of dominating the Web: find a strong signal (the social graph), curate it at scale (the Facebook platform), and you may become the most important company in the Internet economy (the jury is out on Facebook overtaking Google for the crown, but I'd say deliberations are certainly keeping big G up at night).

But a funny thing has started to happen to Facebook - at least for me, and a lot of other folks as well. It's getting to be a pretty noisy place. The problem is one, again, of scale: the more friends I have, the more noise there is, and the less valuable the service becomes. Not to mention the issue of instrumentation: Facebook is a great place for me to instrument my friend graph, but what about my interests, my professional life, and my various other contextual identities? Not to mention, Facebook wasn't a very lively place to discover what's up, at least not until the newsfeed was forced onto the home page.

Credit Twitter for that move. Twitter's original differentiation was its ability to deliver a signal of "what's happening". Facebook quickly followed suit, but Twitter remains the strongest signal, in the main because of its asymmetrical approach to following, as opposed to symmetric friending. Twitter is yet another company that has the potential to be "the next Yahoo or Google" when it comes to signal, discovery, and curation, but it's not there yet. Far too many folks find Twitter to be mostly noise and very little signal.

In its early years, things were even worse. When I first started using Twitter, I wrote quite a bit about Twitter's discovery problem - it was near impossible to find the right folks to follow, and once you did, it was almost as difficult to curate value from the stream of tweets those people created.

Twitter's first answer to its discovery problem - the Suggested User List - was pretty much Yahoo 1994: A subjective, curated list of interesting tweeters. The company's second attempt, "Who To Follow," is a mashup of Google 2001 and Facebook 2007: an algorithm that looks at what content is consumed and who your follow, then suggests folks to follow. I find this new iteration very useful, and have begun to follow a lot more folks because of it.

But now I have a new discovery problem: There's simply too much content for me to grok. (For more on this, see Twitter's Great Big Problem Is Its Massive Opportunity). Add in Facebook (people) and Google search (a proxy for everything on the Web), and I'm overwhelmed by choices, all of them possibly good, but none of them ranked in a way that helps me determine which I should pay attention to, when, or why.

It's 1999 all over again, and I'm not talking about a financing bubble. The ecosystem is ripe for another new player to emerge, and that's one of the reasons I went to see the folks at Tumblr yesterday.

As I pointed out in Social Editors and Super Nodes - An Appreciation of RSS, Tumblr is growing like, well, Google in 2002, Facebook in 2006, or Twitter in 2008. The question I'd like to know is....why?

I'm just starting to play with the service, but I've got a thesis: Tumblr combines the best of self expression (Facebook and blogging platforms) with the best of curation (Twitter and RSS), and seems to have stumbled into a second-order social interest graph to boot (I'm still figuring out the social mores of Tumblr, but I am quite certain they exist). People who use Tumblr a lot tell me it "makes them feel smarter" about what matters in the Web, because it unpacks all sorts of valuable pieces of content into one curated stream - a stream curated by people who you find interesting. It's sort of a rich media Twitter, but the stuff folks are curating seems far more considered, because they are in a more advanced social relationship with their audience than with folks on Twitter. In a way, it feels like the early days of blogging, crossed with the early days of Twitter. With a better CMS and a dash of social networking, and a twist. If that makes any sense at all.

Tumblr, in any case, has its drawbacks: It feels a bit like a walled garden, it doesn't seem to play nice with the "rest of the Web" yet, and - here's the kicker - finding people to follow is utterly useless, at least in the beginning.

Just as with Twitter in the early days, it's nearly impossible to find interesting people to follow on Tumblr, even if you know they're there. For example, I knew that Fred Wilson, who I respect greatly, is a Tumblr user (and investor), so as soon as I joined the service, I typed his name into the search bar at the top of Tumblr's "dashboard" home page. No results. That's because that search bar only searches what's on your page, not all of Tumblr itself. In short, Tumblr's search is deeply broken, just like Twitter's search was back in the day (and Web search was before Google). I remember asking Evan Williams, in 2008, the best way to find someone on Twitter, and his response was "Google them, and add the word Twitter." I'm pretty sure the same is true at present for Tumblr. (It's how I found Fred, anyway).

Continuing the echoes of past approaches to the same problem, Tumblr currently provides a "suggested users" like directory on its site, highlighting folks you might find interesting. I predict this will not be around for long - because it simply doesn't solve the problem we want it to solve. I want to find the right users for me to follow, not ones that folks at Tumblr find interesting.

If Tumblr can iron out these early kinks, well, I'd warrant it will take its place in the pantheon of companies who have found a signal, curated it at scale, and solved yet another important discovery problem. The funny thing is, all of them are still in the game - even Yahoo, who I've spent quite a bit of time with over the past few months. I'm looking forwarding to continuing the conversation about how they approach the opportunity of discovery, and how each might push into new territories. Twitter, for example, seems clearly headed toward a Tumblr-like approach to content curation and discovery with its right hand pane. Google continues to try to solve for people discovery, and Facebook has yet to prove it can scale as a true content-discovery engine.

The folks at Google used to always say "search is a problem that is only five-percent solved." I think now they might really mean "discovery is a problem that will always need to be solved." Keep trying, folks. It gets more interesting by the day.

---

* I'm going to leave out the signals of commerce (What I want to buy) and location (Where I am now) for later ruminations. If you want my early framing thoughts, check out Database of Intentions Chart - Version 2, Updated for Commerce, The Gap Scenario,and My Location Is A Box of Cereal for starters.

This post originally appeared at Searchblog and is republished here with permission.

Join the conversation about this story »

See Also: