In The Flow Past Web Kevin Marks suggests that it’s important that our understanding of the time horizon around the real time web takes in both the flow of real time web streams and their past, the history in those flows.

Our lives in the real world are made up of real time flows around our senses – verbal, activity, visual, aural flows. When we experience them, we only remember some of them. Probably most of them are forgotten; those we do remember are unconsciously synthesized into memory, in what Roger Schank in Tell me a Story: Narrative and Intellegence calls story skeletons or gists. These can be retrieved later if needed, if they are stored in our memory with enough indices that we can find the appropriate one for the occasion.

In commentary about the real time web there seems to be a natural underlying feeling that the closer the real time web gets to replicating real life communication the better. On that assumption then, and drawing the two ideas above together, I’ve been wondering about how, or if, the real time web should be able to forget.

It’s fairly easy to see that search fulfills the remembering aspect in the analogy. If content on the the web is tagged with enough indices, like our brain subconsciously tags experiences, then information we or others need can be retrieved by search engines. This fits in with the insistence – Scoble‘s evangalising of FriendFeed for its advanced search (now to become Facebook’s) comes to mind – that real time services are much more valuable if their history can be comprehensively searched.

But what about forgetting? In real life forgetting is a way of filtering out what is not important, or that which we don’t care to remember. We act on the assumption that much of what we say and do will not be remembered either by us or others. Forgetting seems to me to be both a filter to enable us to remember what is important, and a kind of safety valve. Our heads would likely burst if we were able to search and remember everything.

What we broadcast online is also subject to our normal subconscious forgetting: we forget a lot of what we put online over time, and we can assume our readers forget what we have done too, if it’s not particularly important. We can also be active in forgetting, in the sense that the web is fluid and we can revise, update and delete, as long as we have control over our own data.

On the static web, outdated versions of pages slip away easily.  Doc Searls recently made the point in Because advertising encourages Altzheimer’s that traditional search could, but doesn’t, favour archiving of the sort that enables delving into past versions of what has been online, because it is geared towards advertising not research. On the live web, Scott Rosenberg points out that blogs are potent because they are good at both modes of flow and past, in updates and archives. These are both in our control.-

In real time flow services we can delete or hide individual updates (but only to a certain extent), whole accounts, or choose to make our accounts private. However, we don’t yet have the open unwalled services that would give us the same control over remembering and forgetting conversations that we can have with static web pages and blogs.

But moving back to the more metaphysical questions about remembering, forgetting and the real time flow, in The Real Time Search Dilemma: Consciousness Versus Memory Erick Schonfeld says:

If real time data streams are akin to the living consciousness of the Web, how do you search them? How do you search consciousness? It is not the same as searching memory, which is what Google does when it looks at its indexed archive of the Web and how those pieces of information build up authority over time. The real time search dilemma centers precisely around how to rank results, and how to resolve the tension between recency and relevancy….

Can you search consciousness, or can you only watch it pass by?… But it is clear that in order to make sense of the stream, it needs to be ranked by order of importance as well as by time.

He attributes his thoughts to a conversation with Edo Segal:

“Real time taps into consciousness,” says Segal, “search taps into memory. That is why it so potent. You experience the world in real time.”

Segal gives a deeper explanation in the comments, describing real time search as a leap in our evolution:

Imagine that every public discussion, post, video, event, are all being analyzed at once by an intelligent system, an evolving AI, take a god’s eye view, think of the opportunities. Its not about getting an alert for a stock you follow or a blog post, its about connecting to humanities collective consciousness as these technologies manifest as a real world vibrant avatar of us all.

I’ve been musing about a kind of noosphere like this for some months now, one without religious connotations. If we are implementing platforms and feeding services on the web that approach mimicking functions of the brain,  are we building a world-fed brain-like sphere of knowledge?

Schank relates intelligence to telling and listening to our everyday stories*,  “the process of creating the story also creates the memory structure that will contain the gist of the story for the rest of our lives. Talking is remembering”.  We are defined by the stories we tell. What we don’t tell is forgotten, what is only told seldom is usually forgotten over time, and what we tell over and over is remembered more, but is subject to embroidery and selected emphasis for different effects and audiences, so that truth is at best ephemeral.

If the impetus on the web is for real time flows to be like spontaneous real life conversations – telling stories – and their past is an important dimension too, and can be stored with indices and able to be searched like memory, then the analogy suggests the system itself probably needs to be able to forget, too, and only remember the things that get told with frequency over time. I guess an alternative view is that if the system can remember and retrieve more than people can, it is an advantage, just like Google can be seen as an extension to memory, a kind of Harry Potter pensieve. I’m not sure; I think I’d prefer a system where real time conversations are forgotten unless actively remembered.

*Incidentally, I think this is the secret to why Twitter and microblogging works, but that is for another post.