Domain Model: Conversations

I spend many hours each week tracking conversations. Not the conversations like "hi how are you ....I am fine....", I am talking about conversations that call for actions. They can be actions on my part or actions that i am requesting of others. ..... Most of those conversations have the same pattern centered around making a request, or making a promise and tracking its completion. 

Ideally I would like to track the state of these conversations and many of these conversations occur in email and some occur in IM so any tool I write for managing conversations would reflect that common pattern.

For the academics out there the known model of interaction that encompasses my day to day conversations is based on something called "Speech Action" or "Language Action Perspective". A commercial tool (The Coordinator) has been built using it and seems to have a great deal to recommend it.  The basic ideas behind the Language/Action Perspective:

All human results are achieved through conversations (typically written and spoken)that invoke actions.  Most of us don't need to track conversations that don't lead to actions. Those conversations typically involve exchanges of information  .... 

 

Conversations that lead to actions usually require being tracked....and are effectively a dance...

Those conversations start with either a request for an action or an offer of an action. They can also start off with a "What If" type of conversation such as "Wouldn't it be cool if" and then result in requests or offers of action.

People can accept a request, decline a request, or make a counter offer).

Accepting a request typically involves making a promise. The promise can be completed or revoked.

At some point the conversation for action is complete.

So a tool for  tracking and managing  conversations would allow the following...

1. Opening a Conversation

1a.Generating a Request OR 
1b. Generating an Offer OR
1c. Starting a conversation for a Possibility.

2. Responding 

2a. Accept a request OR

2b. Decline a request OR

2c. Make a counteroffer to a request OR

2d. Comment on or annotate the conversation  OR

2e. Acknowledge someone

2f. Putting off a decision ( i.e. giving a promise by when you will respond)

2g. Make an interim report

2e. Report that the conversation and actions are complete

3. Track conversations (and their participants progress) by their status:

New conversations 

    Conversations missing my response

    Conversations missing other's response

My Promises 

My Requests

Commitments

This may seem incredibly structured , or even artificial, but if you look at the conversations that go into causing projects you will see that they all fall into this structure. Of course, most users would prefer not to use this format for all conversations (or even most of them).  So this should be something optional, where you can did note an e-mail thread as a conversation for action and then track the status of the conversation from that point forward.

What activities should a PIM support

When I first started this blog I naïvely thought that because I have been thinking about this for so long that I had it all firmly clear in my head, and it would pretty much come out in some sensible order.  

Instead I have discovered that along the way I will end up going back and documenting something that seems so self-evident that I have forgotten about it completely.

One example of that is really documenting what are the activities I expect to be able to do with the PIM. This is implicit in a lot of the previous posts, but it is definitely worth making it explicit here.

Basic activities that I expect a user to be able to do include the following:

Capturing and evaluating input: Being able to triage and classify events, tasks, contact information, and other references and notes as they come in. Being able to quickly decide what needs to be handled immediately, what can wait, and if something needs to be scheduled immediately the user should be able to do that quickly even if it means classifying it or scheduling it approximately. "oh look, a dinner invitation, I don't even need to look at it until next week,  so I will throw it into next week.". Being able to throw a note down into the system to be converted to an action or a task later.

Organizing: Going beyond the initial triage to the point where tasks and events are being scheduled in time (or at least the order is being set), as well as coordinating with others. Being able to overlay your own taxonomy/ontology/organizational structure onto the raw objects in the domain ( events, tasks, promises, actions, contacts, etc.). Being able to extend the raw objects of the domain so that they can be personalized to carry the information that the user needs.

Collaborating: Not only sharing calendars, but tracking conversations associated with shared activities.    Tracking the status and progress of projects from organizing your grandmothers birthday party all the way to coordinating the training of a large group of people.

Squirreling things away so the nuts can be found again:  one of the biggest battles for users of a PIM  is storing the data (documents and pictures and other information) associated with an event, a task,a person, or a group in such a way that it can be easily found when needed. 

Finding: Looking through information that you stored off to the side with the intention of finding it again.   Being able to reconstruct timelines based on e-mail conversations and scheduled events. 

Presenting: Visualizing and reporting on the data in a way that is natural for the user. Being able to generate reports on the status of projects from scheduling birthdays and tracking who is coming all the way to larger projects like the implementation of a training regimen for a group of trainers.

Playing with the data model

Hello folks, I am back.

I have been working on the PIM for a bit, but without the space to actually blog about it.

I started experimenting with actually code. I have been working on the basic data model classes and how well they model real-world situations. I am not yet at the point of publishing the code, but I'm starting to get to the point where I can actually think about it.

One of the things I've been working on was trying out the java persistence technology Known as DB 4O. 

My intention has been to find a database storage technology that handled hierarchical data well. Relational databases manage many situations very well but even the best ORM (Object Relational Mapping) solution tends to break down when handling deep, flexible and higly modifiable hierarchies.

In order to handle the kinds of events and extensibility that I am looking for.  I need to be able to store and quickly retrieve objects that are referenced from hierarchies/taxonomies where the objects themselvesstore general attributes.

To give you an example I want to be able to generate taxonomies that have a fairly deep structure ( friends-> college friends->the guys-> the wedding party) and be able to walk that taxonomy and quickly collect all the elements it points to.  Then, if necessary, I would like to be able to rearrange the hierarchy.  This is doable with a relational database and an ORM but no matter how carefully you balance the design, and inevitably there will be performance corner cases that are drastically inconsistent with the performance of the rest of the system. 

In addition, I want users to be able tocreate their own classes of events and tasks. For example, I would like users to be able to create specific events and add their own attributes. if a user would like to distinguish between kids birthday parties and general birthday parties, they should be able to simply create a kid's birthday party event with customized properties and then be able to search within those customized properties.

Generic attributes such as this often present headaches for ORM frameworks, though not nearly as bad as managing hierarchical data.

So I was pretty sold on using a decent object storage framework.  If I could find one! 

DB4O has pretty much answered my requirements.

Through a mixture of byte code instrumentation and good design the DB40 folks have achieved a minimally intrusive persistence framework that effectively speeds up development, unit testing, and general maintainability of the code.

To give you a code example:



// Open the DB

ObjectContainer db;

Configuration configuration = Db4o.newConfiguration();

configuration.generateUUIDs(ConfigScope.GLOBALLY);
configuration.generateVersionNumbers(ConfigScope.GLOBALLY);

db = Db4o.openFile( configuration, filename);

//
// Create a bunch of objects (no matter how deep the hierarchy
//

List items = methodThatCreatesAllTheObjects();

// Store the objects

db.store(items);

// Close the transaction

db.commit();



As you can see there is a minimum of explicit code by default. So I don't have to explicitly open a transaction, that is done when I attempt to store the items. 

There are of course, ways to explicitly do all of the actions necessary, but so far I haven't needed to go outside of the defaults. 

Metadata and Taxonomies (aka How do I find it my way?)

Some of the very best PIMs allow you to organize your data according to your needs. Usually that involves using tags or categories. And a great deal of discussion has gone into whether tagging or categorization is more effective for people who are tracking large amounts of information.

What is the difference? Tagging is like sewing little labels on your socks, categorizing is like picking the specific drawer to pick each type of sock in. I would suggest that the best overview document is Shirkey's document here. He very effectively details the strengths and weaknesses of both methods. I would also suggest Christian Ricci's discussion for very clean examples of solid uses of Taxonomies.

After reading Shirkey's notes you may be left with the idea that Tags are the "One True and Useful Way". To the extent that he talks about (i.e. the world of the web) that may be very true. But in the world of PIM the many weaknesses of Taxonomies become strengths for an individual user. The first person I found that seems to have firmly grasped that is Paul Chen here. And so what I am aiming for is the use of Taxonomies where the user finds it works for them and tags as necessary.

For now I want to talk about Taxonomies (I will expand on tags in another post). First are some definitions. A Taxonomy is a representation of concepts (called Taxa) that have distinct relationships.

For example a Taxonomy of the communities I track in my PIM would have Work, Church, Family, and Friends. Friends would have a distinct subset of "College Friends" and Family would have related subsets for each family grouping for my immediate family, my wife's family and my ex-wife's family (see graphic above).

A taxonomy encodes the relationships between the concepts. These include:

• is-a-type-of relationships like spaghetti is-a-type-of pasta.
• is-an-instance-of-a relationship like "my spaghetti dinner" is-an-instance-of-a "spaghetti dinner".
• is-part-of relationship like Rome is-a-part-of Italy.
• is-associated-with relationship like Spaghetti is-associated-with Italy.

These relationships could be the key to achieving smooth access to the information you need when you need it. For example, if I had a tree of communities associated with each other such as the picture above then when I went to reply to an email from my brother I could quickly be given the oppurtunity to email it to my immediate family or the next highest group up beyond that. If a PIM supported multiple user defined Taxonomies such as a Community Taxonomy or a Calendar Taxonomy then the user could view each level of the taxonomy using the views that made sense for them.

As an example, a taxonomy could have the following levels and relationships Calendar->Jim, Janice, Boys, Church, Work with additional subgroupings of Church->Board of Trustees, Worship Committee. Any type of object could be associated with any level such as Work but only those objects that have an Aspect of LIT would show up in a calendar view or a schedule view. Only those objects with the aspect of LIS would show up in a map view.

Of course, as a user my preference would be for being able to create and manipulate taxonomies quickly as well as associate searches with Taxa for the ability to create flexible taxonomies.

Tailoring the domain model.

In the previous post I stated that I believe the path to a usable PIM lies in allowing the user to tailor the domain model to match how they work. That is a great ideal and very laudable but the world is afloat with software that is so tailorable that it is unusable. How to manage this so that the user is not overwhelmed, annoyed or just plain pissed of is going to take some thought. So the first thing to look at is at what will not vary.

There are some things that I believe form the foundation of the things to be managed in a PIM:

• Events: Things that happen
• Actions: Thing that are done or performed.
• Notes: Things that are recorded for later.
• Contacts: Info on people.

And, after looking at the model for awhile I believe that there are some "aspects" or "potential behaviors" that can be associated with each object. The aspects that I can see immediately are:]

Can be Located in Time abbreviated LIT
We can place this on some sort of chronology (Calendar, Schedule, etc)

Can occupy time abbreviated OT
We can view its duration. This may be a logical part of the LIT aspect above

Can be Located in Space abbreviated LIS
We can place this on some sort of map

Can be tracked abbreviated TRK
We can track its progress

Can have a lifespan abbreviated SPN
We can see the object become relevant and expire.

Much of what determines what is visible in a given context is the aspects associated with the object. And those aspects can be applied or not as necessary. The Chandler project refers to this as "Stamping". For example an Event may be made "mappable" by stamping it as LIS. The user may choose to leave its location indeterminate but it is the stamping of the object as LIS that makes it mappable, not that it is given a location. With these building blocks the user has the potential to create many specialized types as needed. i.e. a Promise could be an Action that can be tracked and has a limited lifespan. In addition, the user can then display a Calendar which contains all things that are LIT whether they be Notes about an Event, an Event or a Promise.

Of course most users will want to filter things down to what they want. But then, now that we have determined what governs what CAN be displayed we come to looking at how we filter things down narrower then the aspects associated with the objects. And for that we need to look at the thing that PIMs are expected to handle well. And that is metadata; taxonomies, tags and categories. Since that is a substantial topic in and of itself, that will be covered in a seperate post.

"You keep using that word. I do not think it means what you think it means."

In my previous post I talked about how often users adapt to the PIM rather than the reverse. And no matter how intelligent the developers are the end result of their designs often seems to come up with eerily similar results (Chandler and Ecco being a somewhat notable exceptions). Most PIMs seem to end up with some variation on Tasks, a Todo List, one or more calendars, some events and some categories....

But the domain that we are all trying to model doesn't seem to be that clean cut. When I started out modeling the domain I discovered that very little appears to be set in stone. Most clear statements I can make about the domain are of the "may" or "can" type.

Here are some of the objects that I believe need to be modeled (in no particular ordering):

Events, Meetings, Appointments, Promises, Tasks, Birthdays, Anniversaries, People, Organizations, Checklists, Todo lists, Schedules, Calendars, Communities, Projects, Conversations, Goals, Journals, Directions, Notes and references to resources.

And here are some example statements I can make about some of these objects:

• Events may be located in time.
• Events may occupy time.
• Events probably will be located in space (have a location)
• Events may have participants
• Events may have an owner

• Actions may have an owner
• Actions may be located in time.
• Actions may occupy time.
• Actions may have a location.
• Actions may have participants.

For almost every object in the domain there is some ambiguity about what its behaviors and attributes are. The ambiguity seems to come from several areas. Some of it comes from the context you are working in at the time: the "Honey-Do" list at home may be managed and tracked differently than the project todo list at work. Some ambiguity comes from the fuzziness of the words themselves: What is the difference between a Task that is scheduled in time and an Event? And some of the ambiguity comes from the precision the user expects from the word itself: One user may consider Tasks very important objects to be managed in priority order with due dates and other users may expect Tasks to be simple entries in a list that you mark off when you are done.

So it seems inevitable that there will be significant mismatches between the model that a PIM uses and the worldview of the user. I think the key thing I see here is that the PIM's domain model needs to be tailorable to match the user's world view.

Where is flexibility needed ?

People have been very generous with their time and talking about how they manage the details of their life using a PIM or equivalent. I have been astounded with how people have adapted to the PIM of their choice.

And that seems to be the theme. Anyone who is using an electronic PIM seems to have done some significant adaptation to the PIM itself. It may be simply accepting that you can only have 15 categories for all Tasks, Events and so on. It may also involve managing things in multiple applications and sharing data between them.

Some of the issues people have adapted to:

• PIMs that have only one calendar.
• PIMs that require you to share the same set of categories among events, contacts, and tasks.
• PIMs that only allow you to set one category for each contact.
• PIMs that assume tasks can only have a due date. If you want to reserve a time for doing it the user needs to schedule it seperately as an event.
• PIMs that don't allow you to schedule an event until everything is known. When and how long , etc..
• PIMs that make it difficult to move the information in and out.
• PIMs that only allow you to track the members of an organization, not the organization itself.

I could go on but the basic idea seems to be that the limitations appear to be arbitrary and sometimes they appear counter intuitive. Yet it seems unlikely to me that everyone involved in the development of a PIM is making arbitrary decisions for the fun of it. There are some very smart people working in this space. Just look at the teams working on Chandler, the KDE PIM or Evolution.

As far as I can tell the issue seems to be how the PIMs choose to manage complexity. All PIMs atempt to manage complexity for the user. Many of them do so by making assumptions about what a user needs.

The tension is between making managing the complexity that gets presented to the user and making the underpinnings flexible enough to support the whole range of activities.

The Target Audience

The Chandler project describes their target users as being "knowledge workers"; people who routinely manipulate bodies of knowledge. My target users are a bit different. The people I am aiming at are the people who have many irons in the fire, who may have many large areas of life that are being managed.

• They have multiple projects, multiple agendas, multiple calendars.
• They have promises to make and track in many areas.
• They have ongoing conversations to be managed.
• They have more than one team that they are tracking or dealing with.
• They're often sharing information with small 2 to 50 person sized groups

This doesn't mean that this will not be of use to people with less diverse and busy lives. But my goal is to make that kind of lifestyle instantly manageable.

Things that work.

Before I gave you this list of what didn't work for me from the existing applications. Now I would like to give you an idea of what I have found that seems to work well or has promise for the future.

As I mentioned in the previous post, Chandler appears to have the best thought out model of the domain. Anyone looking at designing anything in the domain should take a very close look at their requirements and analysis of the domain.

The apparently defunct Aethera project (it looks like they stopped work on it in 2005) introduced me to sub-tasks which I think are a very good idea. TaskCoach also seems to have looked seriously at the subtask issue.

The discontinued ECCO tool comes close to how I want to handle metadata. My wife introduced me to its ability to link ad hoc metadata to presentation or reports in a way that makes information about a schedule accessible at the drop of a hat. I think that the handling of tagging and categorization will be critical to creating an application worth switching to.

There was a DOS e-mail client called Coordinator that allowed you to track the state of conversations in e-mail. It took some discipline to use but gave back a large amount of benefit.

Tagging and Categorization

When categories are not under your control, they are tags.

What I mean by that is if you are importing contacts from someone else AND you have no access to the definition of the categories the contacts are, in effect, being brought in with what appears to be ad-hoc tags associated with them.

This also applies if you are importing and exporting to and from another PIM.

Technology Choices

The cross-platform GUI toolkit

There are two technologies that I consider fairly obvious for this work. the first is the cross-platform GUI toolkit. As far as I can tell, there are really only two contenders out there. The first is QT and the other is the Eclipse RCP. I am planning to use the RCP simply because it does much more out-of-the-box and it is written in Java, which I am far more fluent in nowadays than C++ or C ( besides, I never had a great love of tracking memory allocations).

There are some downsides, especially speed of start up and memory footprint.

pros: Well known, decent performance on most systems, designed for exactly this.

cons: A large memory footprint, a serious learning curve, and potentially slow startup.

The persistent storage

The second technology is what to use for the persistent store. This is a little less obvious, because there are more choices and a lot more variation within the choices. To start with the biggest issue is whether or not to use a relational database. relational databases are very good at what they do and have very well known scaling and programming issues. if I decided to go with Hibernate or JPA I would have a wealth of technical support at my fingertips from their very active user base. The downside is that for managing hierarchical metadata the relational database is not a great choice. In effect, I would be using the relational database for something it's not really designed to model. It could certainly be done using Hibernate or JPA but there would be a perceptible cost in design to fit, maintenance time, deployment issues, and possibly performance.

Since the problem I am dealing with deals with relatively small numbers of objects, intricately linked, it makes sense to go with an object database such as db4o. I looked at several, but db4o's defaults fit in well with my intention to keep things as simple as possible.

I am very interested in any thoughts people have about modeling hierarchical relationships with their preferred databases.

pros: Very natural fit for object persistence. Uses annotations and very simple defaults to make things work. Less resource intensive than most relational databases.

cons: Not as well known or as understood as relational databases. It also has a smaller user community.

PIMs, PIMs, Everywhere

Currently I am using Thunderbird with the calendar plugin as my PIM. It has all the basics and is cross platform even though it has none of the other features I want. In addition I can synch it up to scheduleworld so I have a central repository for my contacts and calendars.

Looking at whats out there right now these are the cross platform (web ones are mentioned separately) ones that I have found and why I have ruled them out so far:

1. The apparently defunct Aethera project Clunky to use but solid.

2. The very ambitious Chandler Project Lots of really great analysis of the domain, some great UI ideas, implementation is clunky, slow and not ready for prime time.

3. Citadel. Server side implementation with Web interface. Very complete and solid in many areas.

4. Haystack Seems interesting but I could never keep the semantic concepts from intruding on actually getting work done. I recently went back to the site and was unable to find the Haystack application to see if it had improved.

5. KDE PIM Toolset I have a soft spot in my heart for the KDE Pim tools. Many of them do things well and quickly and have the most common operations eassily accesisble. But again what they are modeling is a less comprehensive model then I am looking for and the metadata model is very limited.

6. NomadPIM This does what it does very well and has all the basics. It even allows tasks to be categorized easily. The use of special views and perspectives is very utilitarian. The base domain model is again a very small subset of what I am looking for.

7. TaskCoach Some great Ideas and UI concepts. Unfortunately it is really only focused on task management.

There are more Web PIMs (inherently multiplatform) then cross platform
standalone applications. Unfortunately I haven't seen any that allow for
disconnected operation. I don't want my schedule to be held hostage by the
vagaries of net connections.

I would recommend that anyone looking at what is current start with:

http://en.wikipedia.org/wiki/List_of_personal_information_managers

Why this blog.

Simplex
1. Simple or single
2. uni-directional

Saltation
1. The act of leaping, jumping, or dancing.
2. Discontinuous movement, transition, or development; advancement by leaps.

I ended up starting this blog as a clearing house/focal point for
ideas for a PIM (Personal Information Manager) I am writing.

Your reaction may be somewhat underwhelmed. I don't blame you. The net is littered with the bodies of half finished PIMs, each created by someone who wasn't satisfied with the current state-of-the-art.

The bottom line for me is that the current PIMs that I have looked at don't do what I want, and where I want and when I want. And I have looked. I spent 2 years looking and trying out demo versions, etc. No luck. It got to the point where my wife finally asked

When are you going to admit you are going to have to write it?

I did resist the temptation. Really, truly. I didn't want to be known as an engineer that succumbed to the "not invented here" syndrome.

But finally, I gave in.

So what are the things that I need that I have not found so far?

1. It has to run on multiple platforms. I use both Windows and Linux routinely and I don't see that changing anytime soon. This rules out a large number of Windows only applications.

2. It has to support multiple calendars but a single set of events. An
example of this is the eight schedules I maintain at home. I have a calendar for myself, a calendar for my wife, my kids calendar, my ex-wife's calendar, my work calendar, and several calendars for organizations that I donate my time to. If there is an event on my kids calendar that I plan to go to I would like to be able to simply put it on my calendar and then when I move the time of the event on one calendar it shows up correctly moved on the other.

3. It has to support scripting and template of events. I would like to be able to do something like schedule myself to lead an event that I have done several times before and automatically have items added to my task list and time scheduled in the weeks to come for preparation. In fact it's scheduling backwards from the point of the event.

4. It has to support the ability to both tag (assign ad hoc metadata to an object) as well as categorize things in hierarchical user-defined taxonomies.

5. The relationships between communities, events, notes, and tasks should be recordable and navigable. For example, I should be able to associate Projects with communities so that when I am working on a task in that project I have easy/instant access to the e-mail addresses of that associated community.

6. It has to be able to track conversations. For example, if I am heading up a team and I make a request I would like to be able to track and a glance and know how many people have responded, how many have accepted the request, how many have declined the request, and maybe even how many have come back with an alternative suggestion.

7. It has to be able to synch to mobile devices and web services (google, scheduleworld, etc..)

8. Free or low cost (< $150)

Last of all, the user interface should be fast and pertinent. It should be easy to quickly capture information from a user so that it can be triaged later. Easy things should be trivial to do, difficult things should be easy.

So far, I have not found what I'm looking for. The Chandler project comes the closest. I think that the Chandler project has produced the quintessential domain data model for a PIM. Unfortunately, I find that personally, the user interface renders many tasks unusable or unintuitive. In addition I find that how I model the process of working with the data seems antithetical to the Chandler model.

That's all for now. Please feel free to respond with comments. I would love to hear about PIMs that you think meet these criteria. It is quite possible that I missed the PIM I am looking for.