Provided by: libmapi-dev_2.0-3_amd64 
NAME
mapistore-documentation - .PP
Contents
• Revision History
• 1. Introduction
• 1.1. Purpose and Scope
• 1.2. General Overview
• 2. Technical / MAPI Considerations
• 2.1. MAPI objects
• 2.2. MAPI tables
• 2.3. MAPI properties and mapping
• 2.4. Named properties vs known properties
• 3. MAPIStore architecture
• 3.1. INTERFACE layer
• 3.2. PROCESSING layer
• 3.3. BACKENDS layer
• 3.4. Relationship to OpenChange Dispatcher database
• 3.5. Object mapping
• 4. MAPIStore API
• 4.1. Initialization
• 4.2. Backend contexts
• 5. FSOCPF backend
• 5.1. Definition
• 5.2. Namespace and Attributes
• 5.3. Overview
• 5.4. Documentation and References
Revision History
Date Revision Number Author Revision Content 21/05/10 0.3 Julien Kerihuel Add API
documentation for initialization, backend connection contexts and add programming samples
21/05/10 0.2 Julien Kerihuel Merge initial Wiki document and add FSOCPF section 20/05/10
0.1 Julien Kerihuel Draft document.
1. Introduction
1.1. Purpose and Scope
MAPIStore is the SAL component of OpenChange server. SAL stands for Storage Abstraction
Layer. It is the component used by OpenChange Server to push/get information (messages,
folders) to/from storage backends. The following document intends to describe the
overall/theoretical SAL behavior and constraints we need to consider when dealing with
MAPI/EMSMDB. It also describes the semantics and inner working of its storage backends.
1.2. General overview
The main objective of mapistore is to provide an interface layer with a common set of
atomic functions (operations) used to trigger and dispatch data and commands to the
appropriate backend. MAPIStore relies on a backend mechanism specifically designed to
transparently handle some of the MAPI semantics required by any Exchange compatible
server.
The initial idea was to provide to OpenChange a highly customizable storage backend
mechanism which would fit in any situation and any environments. One of the greatest
limitation we have found with existing groupware is the storage layer which is generally
limited to a single solution, service or format and is neither scalable nor modifiable
when user requirements evolve upon time.
MAPIStore solves this problem and go beyond classical limitations. It is not a
revolutionary concept, but the way openchange uses it makes the whole difference and offer
administrators an innovative way to customize storage.
MAPIStore allows you to:
• use a different backend for any top-folder
• transparently move/copy data across backends
• develop new backends quickly
• access all the different backends through an unique API
For example (assuming all associated backends were developed) a user could have the
following storage organization for his mailbox:
• Mails stored using an IMAP backend (Cyrus-IMAP or dovecot)
• Calendar items stored in CalDAV or pushed in Google calendar
• Sent emails and archives/backup stored in a compression backend
• Tasks stored in a MySQL database
• Notes stored on the filesystem
If the user is not satisfied with one of the backend's performance, they would just have
to use an administration tool, change the backend, wait for the replication,
synchronization to finish and there data will be available from the new backend.
Information can be completely decentralized, stored on one of several servers and still be
accessible transparently from OpenChange server.
2. Technical / MAPI Considerations
2.1. MAPI objects
An object is a physical (message, folder) or temporary (table) but generic entity which
can be represented as a 2 columns fixed array with n rows, where each row contains a
property of that entity. One column repesents the property tags (names), and the other
represents the property value (or values).
From a MAPI perspective (network layer), opening an object means:
• opening either a private mailbox or public folder store. These are referenced by EssDN
and not IDs
• opening a message given its PR_MID (Message identifier)
• opening a folder/container given its PR_FID (Folder identifier)
2.2. MAPI tables
Another category of MAPI objects are tables (also known as views) created with MAPI ROPs
such as GetContentsTable, GetHierarchyTable, GetAttachmentTable or GetRulesTable. Views
are temporary representation of the elements that a container holds at a specific moment.
It can be represented as a list of n rows (elements) with n columns (property values):
• GetContentsTables creates a view listing all messages available within a container
• GetHierarchyTable creates a view listing all containers within a container
• GetAttachmentTable creates a view listing all the attachment of a message
• GetRulesTable creates a view listing all permissions associated to a container
Tables are customized through the SetColumns MAPI ROP which will define the property
identifiers to be retrieved. The QueryRows MAPI ROP can then be called to sequentially
retrieve rows and associated property values. A table is similar to a MAPI object except
it is virtual, created on demand to represent a list of objects rather than a unique
object.
2.3. MAPI properties and mapping
There is a large set of fixed and known MAPI properties available. If appropriate backends
are developed, there can be a 1-1 mapping between MAPI properties and backend properties
for some of them. This mapping may even be enough for common purposes. However there will
still be a set of MAPI properties which won't fit in this process.
There are different way to workaround this issue. For example, Kolab is using a Cyrus/IMAP
backend and associate/store MAPI properties to the message using ANNOTATE-MORE within a
XML file format.
OpenChange provides similar mechanism with its OCPF file format.
2.4. Named properties vs known properties
OpenChange server needs to support named properties. An initial set of named properties
can be defined at provisioning time, but this list must not be static. Users must be able
to add, delete, change new entries and create their own set of custom named properties as
required.
3. MAPIStore Architecture
Given that objects representation are similar to SQL database records, an architecture
like the sqlite one makes sense for our purpose:
Component Brief description INTERFACE convenient top-level functions (Public API)
accessed through EMSMDB providers PROCESSING set of atomic operations (open, read, write,
close, mkdir, rmdir etc.) BACKENDS The code which does things in the specified storage
system (mysql, fsocpf, imap etc.)
3.1. INTERFACE layer
The interface layer doesn't have any knowledge about mapistore internals, how or where
objects are stored. The interface uses MAPI data structure, supplies PR_FID and PR_MID
values and assumes the interface layer will return a pack of data it can directly use
without further or significant modifications. The interface layer functions should also
have (as a parameter) a pointer to a mapistore context with private/opaque set of
information (void *) about the object.
3.2. PROCESSING layer
The processing layer is responsible for:
• mapping OpenChange objects identifiers (PR_FID, PR_MID) to unique backends object
identifiers (on purpose, depending on the kind of backend).
• format input/output data: glue between INTERFACE and BACKENDS
• relay input requests to the correct backend through atomic operations
• maintain mapistore's integrity
3.3. BACKENDS layer
The backends layer has a list of modules identified at mapistore initialization and
available across user sessions, which means unique initialization at server start-up. Each
module is a backend (fsocpf, sqlite, imap, etc.) and similarly to many other openchange
components is loaded as a DSO object (dynamic shared object)
3.4. Relationship to OpenChange Dispatcher database
MAPIStore and the openchange 'dispatcher' database (openchange.ldb) are completely
unrelated. MAPIStore is a standalone API and developers can use it independently from
OpenChange server.
However, the mapistore API has initially been designed to be used by OpenChange server,
and OpenChange server is using a tiny indexing database which describes user mailboxes top
level containers. In openchange.ldb the mapistore_uri attribute is attached to top level
containers and its value points to a valid mapistore URI (namespace + path). Note that a
single user can have several different types of mapistore databases in use (one for each
of the top level containers).
The is the only relationship between the database and the store: The database points to
store locations.
3.5. Object mapping
MAPIStore needs to maintain a hash database linking unique OpenChange identifiers to
unique backend identifiers. This hash table can be stored within a TDB database.
MAPIStore is responsible for managing IDs mapping between OpenChange objects and backend
specific objects. It maintains a list of free identifiers which it reallocates on demand
whenever a backend needs (mapistore_register_id()) one or where it wants to release one
(mapistore_unregister_id()).
4. MAPIStore API
MAPIStore relies on the talloc library for memory allocation.
4.1. Initialization
If there was a 'hello mapistore' program, it would only require to make 2 calls:
• mapistore_init:
The initialization routine initializes the mapistore general context used along all
mapistore calls, the mapping context databases (described below) and finally load all
the backends available (DSO). When this operation is successful, developers are ready to
make mapistore calls.
• mapistore_release:
The release operation uninitializes the mapistore general context, closes connections to
database and frees the remaining allocated memory.
mapistore_sample1.c
#include <mapistore/mapistore.h>
int main(int ac, const char *av[])
{
TALLOC_CTX *mem_ctx;
struct mapistore_context *mstore_ctx;
int retval;
/* Step 1. Create the talloc memory context */
mem_ctx = talloc_named(NULL, 0, 'mapistore_sample1');
/* Step 2. Initialize mapistore system */
mstore_ctx = mapistore_init(mem_ctx, NULL);
if (!mstore_ctx) {
exit (1);
}
/* Step 3. Uninitialize mapistore system */
retval = mapistore_release(mstore_ctx);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
return 0;
}
$ export PKG_CONFIG_PATH=/usr/local/samba/lib/pkgconfig
$ gcc mapistore_sample1.c -o mapistore_sample1 `pkg-config --cflags --libs libmapistore`
$ ./mapistore_sample1
$
4.2. Backend contexts
MAPIStore registers and loads its backends upon initialization. It means they are only
instantiated/initialized one time during the whole server lifetime and the same code is
used for all users and all mapistore folders.
These backend contexts (or connection contexts) are identified by a context id, which is
an unsigned 32 bit integer which references the context during its lifetime. If OpenChange
is used in a very large environment with many top folders (which implies the same number
of mapistore contexts), or if OpenChange server has an incredibly long uptime, it would be
possible to run out of available context identifiers.
In order to prevent this situation from happening, mapistore implements context databases
where it stores available/free/used context identifiers:
• mapistore_id_mapping_used.tdb: TDB database with used IDs
• mapistore_id_mapping_free.tdb: TDB database with available pool of IDs
MAPIStore provides a convenient set of functions to manage backend contexts:
• mapistore_set_mapping_path: Defines the path where context databases are stored. Call to
this function is optional and default path would be used instead. However if a call to
this function has to be made, it must be done before any call to mapistore (even
mapistore_init).
• mapistore_add_context: Add a new connection context to mapistore
• mapistore_del_context: Delete a connection context from mapistore
mapistore_sample2.c:
#include <mapistore/mapistore.h>
int main(int ac, const char *av[])
{
TALLOC_CTX *mem_ctx;
struct mapistore_context *mstore_ctx;
int retval;
uint32_t context_id = 0;
uint32_t context_id2 = 0;
/* Step 1. Create the talloc memory context */
mem_ctx = talloc_named(NULL, 0, 'mapistore_sample1');
/* Step 2. Set the mapping path to /tmp */
retval = mapistore_set_mapping_path('/tmp');
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
/* Step 3. Initialize mapistore system */
mstore_ctx = mapistore_init(mem_ctx, NULL);
if (!mstore_ctx) {
exit (1);
}
/* Step 4. Add connection contexts */
retval = mapistore_add_context(mstore_ctx, 'fsocpf:///tmp/Inbox', &context_id);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
retval = mapistore_add_context(mstore_ctx, 'fsocpf:///tmp/Sent Items', &context_id2);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
/* Step 5. Release connection contexts */
retval = mapistore_del_context(mstore_ctx, context_id);
retval = mapistore_del_context(mstore_ctx, context_id2);
/* Step 6. Uninitialize mapistore system */
retval = mapistore_release(mstore_ctx);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
return 0;
}
$ ./mapistore_sample2
sqlite3 backend initialized
fsocpf backend initialized
namespace is fsocpf:// and backend_uri is '/tmp/Inbox'
[fsocpf_create_context:49]
namespace is fsocpf:// and backend_uri is '/tmp/Sent Items'
[fsocpf_create_context:49]
$
5. FSOCPF Backend
5.1. Definition
FSOCPF stands for FileSystem and OpenChange Property Files. It is a backend designed to
help developers testing OpenChange server code easily. The main idea is to have a backend
we can manipulate, analyze and modify from the Linux console without having to develop a
specific tool. This backend uses the UNIX filesystem for folder semantics and the OCPF
file format as a way to store MAPI objects easily.
5.2. Namespace and Attributes
The namespace for this backend is:
fsocpf://
The mapistore_uri attribute for the folder definition in openchange.ldb must be a valid
path where the last part of the URI is the FSOCPF container folder to create on the
filesystem.
5.3. Overview
[+] Private user storage space
|
+-[+] Top-MAPIStore folder (Inbox)
|
+-[+] 0xf1000001 (mapistore folder1)
| |
| +-[+] .properties (OCPF)
| |
| +-[+] 0xe10000001.ocpf (message - OCPF)
| |
| +-[+] 0xe10000001 (attachment folder)
| |
| +-[+] 1.ocpf (PR_ATTACH_NUM)
| |
| +-[+] 1.data (attachment / stream data)
|
+-[+] 0xf2000001 (mapistore folder2)
|
+-[+] .properties (OCPF)
The figure above exposes the storage architecture of the FSOCPF backend using a real-world
example. In this use case, we have decided to associate the FSOCPF backend to the Inbox
folder. It means that any folder created under Inbox or any message stored within Inbox at
any level of depth is stored and retrieved from the path defined in openchange.ldb.
In openchange.ldb, the mapistore_uri attribute of the Inbox record points to:
fsocpf://Private user storage space/Inbox
where Private user storage space can for example be
/usr/local/samba/private/mapistore/$username
Under Inbox, we have created 2 folders:
• 0xf1000001 folder1
• 0xf2000001 folder2
These folders are identified/named using their FID. Since they are classical filesystem
folders, we can't associate attributes to them such as a folder comment, or the container
class. All these attributes with the displayable name are stored into the .properties file
within the folder.
Inside 0xf1000001 folder, we have 1 message named 0xe10000001.ocpf stored in the OCPF file
format (property/value pair). Any properties associated to the message (subject,
recipient, body) are stored within this file.
This message also has attachments. Attachments are stored within a directory at the same
level named 0xe10000001 (message name without OCPF extension). Within this directory, we
find the attachments named using the PR_ATTACH_NUM property value and the OCPF file
extension. The content of the attachment is stored in $PR_ATTACH_NUM.data - in this case
1.data.
5.4. Documentation and References
• OpenChange Property File format (OCPF) documentation