Flexible Storage System (FS2)

Why FS2?

There are similar technologies out there such as Apache Commons Virtual File System (VFS) and Apache Jackrabbit. FS2 distinguishes itself with the following:

• FS2 Objects are relatively simple, having a URI, headers, and a blob. This makes it a natural fit to map and handle web requests.
• Storage-agnostic API. Client code does not know about the underlying persistence store. For example, a VFS URL make look like jar:/a/b, where fs2 is simply fs2:/a/b/.
• TDD friendly. Use the default FS2 repo (in-memory) while developing for easy testing without minding the complexities of database/filesystem stores. Then when the code is ready for prime time, simply flip a switch (ie change "mem" to "mongo"), and objects will be persisted.
• Lightweight dependencies. The core FS2 API code is lightweight, and for any given deployment scenario, you need only to include the concrete repository that will be used.
• Built-in tests. It's easy to have confidence in a new concrete repository implementation when you can plug it right into an existing test framework.
• Less config. By default, FS2 stores objects in memory, and there is zero configuration required.
• Easier config. FS2 is not going to require heaps of XML files defining factories in order to work. Just override the default values you wish to change, in code or by providing a properties file in json format.
• Leaves the typical "heavyweightness" of Java frameworks behind. IE uses json for config and object descriptors, relies on default values so the only existing configuration is override configuration.

Interacting with the API

FS2 decouples meta-information like object size, date created, compression, etc from the object itself. The most common meta-data. such as date created, is modeled within the meta class. Extended attributes are contained in unmodeled key value pairs called headers. Client code can interact with the FS2 api using the meta object, or URI. Here is a simple CRUD example.

Basic Crud

// get an instance of fs2 with default properties
FlexibleStorageSystem FS2 = FS2Factory.newInstance();

// create the object.  initially this object will have no data associated with it, just a name and some initial metadata.
FS2ObjectMeta foo = FS2.createObjectEntry("/foo");
    
// add a custom header field to object foo
FS2.addHeader(foo, "isText", "true");
    
// add some contents to foo
InputStream is = new ByteArrayInputStream("hello world".getBytes());
FS2.writePayloadFromStream(foo, is);
    
// delete
FS2.delete(foo);

FS2 also supports the hierarchical arrangement of objects, like you would see in a filesystem, or any tree like data structure.

• In the code below, we create four distinct objects, all belonging to a tree with root /foo
  • This is a convenience method, and objects can just as easily be created one by one.
  • The nodes are created in order of the parameter list.
  • Any nodes in a path that do not previously exist will be created implicitly.
  • If a specified node pre-exists, an exception will be thrown.
    • For example, if the first two parameters below were switched, create "/foo" will fail since it was implicitly created when create "/foo/bar" was executed.
  • The (meta) objects that were explicitly created will be returned by the method.
• Then we get the reference to bar's metadata by accessing the returned array.
  • Alternatively, we can access bar by providing its uri directly to FS2. Note this means an additional fetch.
• Children and descendants can be listed with or without regular expression filters.
• To delete a node with children, use deleteRecursive() or an exception will be thrown.

Working with trees

// create five nodes
FS2ObjectMeta[] nodes = FS2.createObjectEntries("/foo", "/foo/bar", "/foo/baz", "/foo/bar/bam", "/foo/bar/moo");

// can access meta like this
FS2ObjectMeta foo = nodes[0];
FS2ObjectMeta bar = nodes[1];

// sanity check a node exists by refetching from fs2.  note bar2 should equal() bar unless another thread updated (ie) headers in between create and fetch  
FS2ObjectMeta bar2 = FS2.fetchObject("/foo/bar");

// list descendants of foo who's names begin with "m" (expect moo) 
FS2.listDescendants(foo, "*/m.*");
    
// delete bar and bar/bam
FS2.deleteRecursive(bar);

// get foo's remaining descendants.  (expect just baz)
FS2.listDescendants(nodes[0]);

Data migration

Migrating from one storage provider to another can be done with just a few lines of code. It requires obtaining two instances of fs2, one for each provider, and making a move call. Below is an example of migrating from the file storage provider to a Mongo-backed storage provider.

// copy all contents from the file repo to mongoDB
FlexibleStorageSystem fileRepo = FS2Factory.newInstance("file"); // override default and force an instance backed by the "file" storage provider
FlexibleStorageSystem mongoRepo = FS2Factory.newInstance("mongo"); // override default and force an instance backed by the "mongo" storage provider
fileRepo.copyTo(mongoRepo);