trusty (3) mapistore-documentation.3.gz
NAME
mapistore-documentation - .PP
Contents
• Revision History
• 1. Introduction
• 1.1. Purpose and Scope
• 1.2. General Overview
• 2. Technical / MAPI Considerations
• 2.1. MAPI objects
• 2.2. MAPI tables
• 2.3. MAPI properties and mapping
• 2.4. Named properties vs known properties
• 3. MAPIStore architecture
• 3.1. INTERFACE layer
• 3.2. PROCESSING layer
• 3.3. BACKENDS layer
• 3.4. Relationship to OpenChange Dispatcher database
• 3.5. Object mapping
• 4. MAPIStore API
• 4.1. Initialization
• 4.2. Backend contexts
• 5. FSOCPF backend
• 5.1. Definition
• 5.2. Namespace and Attributes
• 5.3. Overview
• 5.4. Documentation and References
Revision History
Date Revision Number Author Revision Content 21/05/10 0.3 Julien Kerihuel Add API documentation for
initialization, backend connection contexts and add programming samples 21/05/10 0.2 Julien Kerihuel
Merge initial Wiki document and add FSOCPF section 20/05/10 0.1 Julien Kerihuel Draft document.
1. Introduction
1.1. Purpose and Scope
MAPIStore is the SAL component of OpenChange server. SAL stands for Storage Abstraction Layer. It is the
component used by OpenChange Server to push/get information (messages, folders) to/from storage backends.
The following document intends to describe the overall/theoretical SAL behavior and constraints we need
to consider when dealing with MAPI/EMSMDB. It also describes the semantics and inner working of its
storage backends.
1.2. General overview
The main objective of mapistore is to provide an interface layer with a common set of atomic functions
(operations) used to trigger and dispatch data and commands to the appropriate backend. MAPIStore relies
on a backend mechanism specifically designed to transparently handle some of the MAPI semantics required
by any Exchange compatible server.
The initial idea was to provide to OpenChange a highly customizable storage backend mechanism which would
fit in any situation and any environments. One of the greatest limitation we have found with existing
groupware is the storage layer which is generally limited to a single solution, service or format and is
neither scalable nor modifiable when user requirements evolve upon time.
MAPIStore solves this problem and go beyond classical limitations. It is not a revolutionary concept, but
the way openchange uses it makes the whole difference and offer administrators an innovative way to
customize storage.
MAPIStore allows you to:
• use a different backend for any top-folder
• transparently move/copy data across backends
• develop new backends quickly
• access all the different backends through an unique API
For example (assuming all associated backends were developed) a user could have the following storage
organization for his mailbox:
• Mails stored using an IMAP backend (Cyrus-IMAP or dovecot)
• Calendar items stored in CalDAV or pushed in Google calendar
• Sent emails and archives/backup stored in a compression backend
• Tasks stored in a MySQL database
• Notes stored on the filesystem
If the user is not satisfied with one of the backend's performance, they would just have to use an
administration tool, change the backend, wait for the replication, synchronization to finish and there
data will be available from the new backend.
Information can be completely decentralized, stored on one of several servers and still be accessible
transparently from OpenChange server.
2. Technical / MAPI Considerations
2.1. MAPI objects
An object is a physical (message, folder) or temporary (table) but generic entity which can be
represented as a 2 columns fixed array with n rows, where each row contains a property of that entity.
One column repesents the property tags (names), and the other represents the property value (or values).
From a MAPI perspective (network layer), opening an object means:
• opening either a private mailbox or public folder store. These are referenced by EssDN and not IDs
• opening a message given its PR_MID (Message identifier)
• opening a folder/container given its PR_FID (Folder identifier)
2.2. MAPI tables
Another category of MAPI objects are tables (also known as views) created with MAPI ROPs such as
GetContentsTable, GetHierarchyTable, GetAttachmentTable or GetRulesTable. Views are temporary
representation of the elements that a container holds at a specific moment. It can be represented as a
list of n rows (elements) with n columns (property values):
• GetContentsTables creates a view listing all messages available within a container
• GetHierarchyTable creates a view listing all containers within a container
• GetAttachmentTable creates a view listing all the attachment of a message
• GetRulesTable creates a view listing all permissions associated to a container
Tables are customized through the SetColumns MAPI ROP which will define the property identifiers to be
retrieved. The QueryRows MAPI ROP can then be called to sequentially retrieve rows and associated
property values. A table is similar to a MAPI object except it is virtual, created on demand to represent
a list of objects rather than a unique object.
2.3. MAPI properties and mapping
There is a large set of fixed and known MAPI properties available. If appropriate backends are developed,
there can be a 1-1 mapping between MAPI properties and backend properties for some of them. This mapping
may even be enough for common purposes. However there will still be a set of MAPI properties which won't
fit in this process.
There are different way to workaround this issue. For example, Kolab is using a Cyrus/IMAP backend and
associate/store MAPI properties to the message using ANNOTATE-MORE within a XML file format.
OpenChange provides similar mechanism with its OCPF file format.
2.4. Named properties vs known properties
OpenChange server needs to support named properties. An initial set of named properties can be defined at
provisioning time, but this list must not be static. Users must be able to add, delete, change new
entries and create their own set of custom named properties as required.
3. MAPIStore Architecture
Given that objects representation are similar to SQL database records, an architecture like the sqlite
one makes sense for our purpose:
Component Brief description INTERFACE convenient top-level functions (Public API) accessed through
EMSMDB providers PROCESSING set of atomic operations (open, read, write, close, mkdir, rmdir etc.)
BACKENDS The code which does things in the specified storage system (mysql, fsocpf, imap etc.)
3.1. INTERFACE layer
The interface layer doesn't have any knowledge about mapistore internals, how or where objects are
stored. The interface uses MAPI data structure, supplies PR_FID and PR_MID values and assumes the
interface layer will return a pack of data it can directly use without further or significant
modifications. The interface layer functions should also have (as a parameter) a pointer to a mapistore
context with private/opaque set of information (void *) about the object.
3.2. PROCESSING layer
The processing layer is responsible for:
• mapping OpenChange objects identifiers (PR_FID, PR_MID) to unique backends object identifiers (on
purpose, depending on the kind of backend).
• format input/output data: glue between INTERFACE and BACKENDS
• relay input requests to the correct backend through atomic operations
• maintain mapistore's integrity
3.3. BACKENDS layer
The backends layer has a list of modules identified at mapistore initialization and available across user
sessions, which means unique initialization at server start-up. Each module is a backend (fsocpf, sqlite,
imap, etc.) and similarly to many other openchange components is loaded as a DSO object (dynamic shared
object)
3.4. Relationship to OpenChange Dispatcher database
MAPIStore and the openchange 'dispatcher' database (openchange.ldb) are completely unrelated. MAPIStore
is a standalone API and developers can use it independently from OpenChange server.
However, the mapistore API has initially been designed to be used by OpenChange server, and OpenChange
server is using a tiny indexing database which describes user mailboxes top level containers. In
openchange.ldb the mapistore_uri attribute is attached to top level containers and its value points to a
valid mapistore URI (namespace + path). Note that a single user can have several different types of
mapistore databases in use (one for each of the top level containers).
The is the only relationship between the database and the store: The database points to store locations.
3.5. Object mapping
MAPIStore needs to maintain a hash database linking unique OpenChange identifiers to unique backend
identifiers. This hash table can be stored within a TDB database.
MAPIStore is responsible for managing IDs mapping between OpenChange objects and backend specific
objects. It maintains a list of free identifiers which it reallocates on demand whenever a backend needs
(mapistore_register_id()) one or where it wants to release one (mapistore_unregister_id()).
4. MAPIStore API
MAPIStore relies on the talloc library for memory allocation.
4.1. Initialization
If there was a 'hello mapistore' program, it would only require to make 2 calls:
• mapistore_init:
The initialization routine initializes the mapistore general context used along all mapistore calls,
the mapping context databases (described below) and finally load all the backends available (DSO). When
this operation is successful, developers are ready to make mapistore calls.
• mapistore_release:
The release operation uninitializes the mapistore general context, closes connections to database and
frees the remaining allocated memory.
mapistore_sample1.c
#include <mapistore/mapistore.h>
int main(int ac, const char *av[])
{
TALLOC_CTX *mem_ctx;
struct mapistore_context *mstore_ctx;
int retval;
/* Step 1. Create the talloc memory context */
mem_ctx = talloc_named(NULL, 0, 'mapistore_sample1');
/* Step 2. Initialize mapistore system */
mstore_ctx = mapistore_init(mem_ctx, NULL);
if (!mstore_ctx) {
exit (1);
}
/* Step 3. Uninitialize mapistore system */
retval = mapistore_release(mstore_ctx);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
return 0;
}
$ export PKG_CONFIG_PATH=/usr/local/samba/lib/pkgconfig
$ gcc mapistore_sample1.c -o mapistore_sample1 `pkg-config --cflags --libs libmapistore`
$ ./mapistore_sample1
$
4.2. Backend contexts
MAPIStore registers and loads its backends upon initialization. It means they are only
instantiated/initialized one time during the whole server lifetime and the same code is used for all
users and all mapistore folders.
These backend contexts (or connection contexts) are identified by a context id, which is an unsigned 32
bit integer which references the context during its lifetime. If OpenChange is used in a very large
environment with many top folders (which implies the same number of mapistore contexts), or if OpenChange
server has an incredibly long uptime, it would be possible to run out of available context identifiers.
In order to prevent this situation from happening, mapistore implements context databases where it stores
available/free/used context identifiers:
• mapistore_id_mapping_used.tdb: TDB database with used IDs
• mapistore_id_mapping_free.tdb: TDB database with available pool of IDs
MAPIStore provides a convenient set of functions to manage backend contexts:
• mapistore_set_mapping_path: Defines the path where context databases are stored. Call to this function
is optional and default path would be used instead. However if a call to this function has to be made,
it must be done before any call to mapistore (even mapistore_init).
• mapistore_add_context: Add a new connection context to mapistore
• mapistore_del_context: Delete a connection context from mapistore
mapistore_sample2.c:
#include <mapistore/mapistore.h>
int main(int ac, const char *av[])
{
TALLOC_CTX *mem_ctx;
struct mapistore_context *mstore_ctx;
int retval;
uint32_t context_id = 0;
uint32_t context_id2 = 0;
/* Step 1. Create the talloc memory context */
mem_ctx = talloc_named(NULL, 0, 'mapistore_sample1');
/* Step 2. Set the mapping path to /tmp */
retval = mapistore_set_mapping_path('/tmp');
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
/* Step 3. Initialize mapistore system */
mstore_ctx = mapistore_init(mem_ctx, NULL);
if (!mstore_ctx) {
exit (1);
}
/* Step 4. Add connection contexts */
retval = mapistore_add_context(mstore_ctx, 'fsocpf:///tmp/Inbox', &context_id);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
retval = mapistore_add_context(mstore_ctx, 'fsocpf:///tmp/Sent Items', &context_id2);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
/* Step 5. Release connection contexts */
retval = mapistore_del_context(mstore_ctx, context_id);
retval = mapistore_del_context(mstore_ctx, context_id2);
/* Step 6. Uninitialize mapistore system */
retval = mapistore_release(mstore_ctx);
if (retval != MAPISTORE_SUCCESS) {
exit (1);
}
return 0;
}
$ ./mapistore_sample2
sqlite3 backend initialized
fsocpf backend initialized
namespace is fsocpf:// and backend_uri is '/tmp/Inbox'
[fsocpf_create_context:49]
namespace is fsocpf:// and backend_uri is '/tmp/Sent Items'
[fsocpf_create_context:49]
$
5. FSOCPF Backend
5.1. Definition
FSOCPF stands for FileSystem and OpenChange Property Files. It is a backend designed to help developers
testing OpenChange server code easily. The main idea is to have a backend we can manipulate, analyze and
modify from the Linux console without having to develop a specific tool. This backend uses the UNIX
filesystem for folder semantics and the OCPF file format as a way to store MAPI objects easily.
5.2. Namespace and Attributes
The namespace for this backend is:
fsocpf://
The mapistore_uri attribute for the folder definition in openchange.ldb must be a valid path where the
last part of the URI is the FSOCPF container folder to create on the filesystem.
5.3. Overview
[+] Private user storage space
|
+-[+] Top-MAPIStore folder (Inbox)
|
+-[+] 0xf1000001 (mapistore folder1)
| |
| +-[+] .properties (OCPF)
| |
| +-[+] 0xe10000001.ocpf (message - OCPF)
| |
| +-[+] 0xe10000001 (attachment folder)
| |
| +-[+] 1.ocpf (PR_ATTACH_NUM)
| |
| +-[+] 1.data (attachment / stream data)
|
+-[+] 0xf2000001 (mapistore folder2)
|
+-[+] .properties (OCPF)
The figure above exposes the storage architecture of the FSOCPF backend using a real-world example. In
this use case, we have decided to associate the FSOCPF backend to the Inbox folder. It means that any
folder created under Inbox or any message stored within Inbox at any level of depth is stored and
retrieved from the path defined in openchange.ldb.
In openchange.ldb, the mapistore_uri attribute of the Inbox record points to:
fsocpf://Private user storage space/Inbox
where Private user storage space can for example be
/usr/local/samba/private/mapistore/$username
Under Inbox, we have created 2 folders:
• 0xf1000001 folder1
• 0xf2000001 folder2
These folders are identified/named using their FID. Since they are classical filesystem folders, we can't
associate attributes to them such as a folder comment, or the container class. All these attributes with
the displayable name are stored into the .properties file within the folder.
Inside 0xf1000001 folder, we have 1 message named 0xe10000001.ocpf stored in the OCPF file format
(property/value pair). Any properties associated to the message (subject, recipient, body) are stored
within this file.
This message also has attachments. Attachments are stored within a directory at the same level named
0xe10000001 (message name without OCPF extension). Within this directory, we find the attachments named
using the PR_ATTACH_NUM property value and the OCPF file extension. The content of the attachment is
stored in $PR_ATTACH_NUM.data - in this case 1.data.
5.4. Documentation and References
• OpenChange Property File format (OCPF) documentation