Storage

From DreamFactory
Jump to: navigation, search
m
m
 
(5 intermediate revisions by the same user not shown)
Line 4: Line 4:
 
Mounts can be local or remote storage. As long as the storage space is accessible via the file system of the server, it can be utilized for storage.
 
Mounts can be local or remote storage. As long as the storage space is accessible via the file system of the server, it can be utilized for storage.
  
By default, DreamFactory Enterprise&trade; defines the mount point <pre>/data/storage</pre> for your initial cluster. This directory contains a subdirectory for each cluster. The initial installation defaults to '''local''' as the storage area. This exists on your system in <pre>/data/storage/local</pre>. When you create new clusters, you can configure their storage areas in a different subdirectory, or different mount point altogether. It is designed to be flexible and map to most storage scenarios.
+
By default, DreamFactory Enterprise&trade; defines the mount point <code>/data/storage</code> for your initial cluster. This directory contains a subdirectory for each cluster. The initial installation defaults to '''local''' as the storage area. This exists on your system in <code>/data/storage/local</code>. When you create new clusters, you can configure their storage areas in a different subdirectory, or different mount point altogether. It is designed to be flexible and map to most storage scenarios. This is termed the '''root storage path'''.
  
 
== Layout and Structure ==
 
== Layout and Structure ==
Line 11: Line 11:
 
[[File:storage-layout.png]]
 
[[File:storage-layout.png]]
  
The <pre>/data</pre> tree contains self-explanatory subdirectories. Under <pre>/data/storage</pre> is where it begins to get interesting.
+
The <code>/data</code> tree contains self-explanatory subdirectories. Under <code>/data/storage</code> is where it begins to get interesting.
  
=== Lightweight Striping ===
+
=== Data Splaying ===
To ''spread'' the storage load, as it were, of thousands of virtual instances, the instance storage tree implements a simplistic hashed directory tree. Each [[DFE/Dashboard|Dashboard]] user is assigned a unique storage ID in the form of a GUID (<pre>[[DFE/Architecture/Database/User|User.storage_id_text]]</pre>). This GUID is hashed, and with it and the first four bytes are used to create a three-level nested subdirectory. Example:
+
With thousands of virtual instances, you don't want their storage areas all homed in the same directory. Firstly, you'll run out of '''inodes''' eventually. Secondly, as the more and more entries are added to the file system, access slows down considerably. To combat this performance decay, DreamFactory Enterprise builds a multi-layered storage structure using user information. The instance storage tree then becomes a simplistic hashed directory tree. We call this ***Data Splaying***.  
  
User id #1 has a storage ID of <pre>00933DBC-C7A9-DE0E-5D38-FB23B574A580</pre>.  
+
Each [[DFE/Dashboard|Dashboard]] user is assigned a unique storage ID in the form of a GUID (<code>[[DFE/Architecture/Database/User|User.storage_id_text]]</code>). This GUID is hashed, and with it and the first two bytes are used to create a two-level nested subdirectory. Example with a fictitious user #1:
Applying a SHA1 hash gives us <pre>9414f5052eead73c262d5ad89922541426208763ce86c2b64cc6a6680914a915</pre>.
+
  
 +
* User id #1 has a storage ID of <code>00933DBC-C7A9-DE0E-5D38-FB23B574A580</code>.
 +
* Hashing the storage ID gives us <code>9414f5052eead73c262d5ad89922541426208763ce86c2b64cc6a6680914a915</code>.
 +
* From the hash, the first two bytes become the first directory level <code>/94</code>.
 +
* The hash itself is the final component <code>/94/9414f5052eead73c262d5ad89922541426208763ce86c2b64cc6a6680914a915</code>.
  
  
== Preventative Maintenance ==
+
This final result is the ''relative'' path under the '''root storage path''' (which, as stated above, defaults to <code>/data/storage/local</code>). Therefore, the full path to the storage area of user id #1 is <code>/data/storage/local/94/14/9414f5052eead73c262d5ad89922541426208763ce86c2b64cc6a6680914a915</code>. This is termed a user's '''storage root path''' as all instances for this user will reside in this directory. Also in this directory is a private area that survives instance termination. Currently, nothing is stored in this directory.
=== Automated ===
+
 
=== Manual ===
+
== Storage Utilities ==
 +
The DreamFactory Enterprise [https://github.com/dreamfactorysoftware/dfe-storage storage support library] provides a service called '''InstanceStorage'''. It can be used to retrieve instance storage information. A facade is also provided for easy access.
 +
 
 +
=== Usage ===
 +
Adding the storage library to your own applications is simple. Just add the requirement <code lang="json">"dreamfactory/dfe-storage": "1.0.*"</code> to your <code>composer.json</code> file and run <code>composer update</code>. Once installed, add the provider to your <code>config/app.php</code>:
 +
 
 +
<source lang="php">
 +
return [
 +
    ...
 +
    'providers' => [
 +
    ...
 +
        'DreamFactory\Enterprise\Storage\Providers\InstanceStorageServiceProvider::class,
 +
    ...
 +
    ],
 +
    ...
 +
];
 +
</source>
 +
 
 +
=== Available Methods ===
 +
The InstanceStorage facade offers the following useful methods for retrieving instance storage information. These come in two flavors. The first returns the path as a string. The second type returns a [https://github.com/gcampbell/flysystem Flysystem] object which represents the mount point. The <code>$tag</code> parameter is used to identify specific directories in the local cache.
 +
 
 +
==== Path Getters ====
 +
* static string getStoragePath(Instance $instance, $append = null, $create = false)
 +
* static string getPrivatePath(Instance $instance, $append = null, $create = false)
 +
* static string getOwnerPrivatePath(Instance $instance, $append = null, $create = false)
 +
* static string getSnapshotPath(Instance $instance, $append = null, $create = false)
 +
 
 +
==== Mount Getters ====
 +
* static Filesystem getStorageMount(Instance $instance, $tag = 'storage:instance-id')
 +
* static Filesystem getPrivateStorageMount(Instance $instance, $tag = 'private:instance-id')
 +
* static Filesystem getOwnerPrivateStorageMount(Instance $instance, $tag = 'owner-private:instance-id')
 +
* static Filesystem getSnapshotMount(Instance $instance, $tag = 'snapshot:instance-id')

Latest revision as of 23:27, 14 December 2015

Instance Storage

When DreamFactory Enterprise™ provisions a new instance, a directory is created for it to use as local storage. The location of this directory depends on the mount assigned to the server on which the instance was provisioned.

Mounts can be local or remote storage. As long as the storage space is accessible via the file system of the server, it can be utilized for storage.

By default, DreamFactory Enterprise™ defines the mount point /data/storage for your initial cluster. This directory contains a subdirectory for each cluster. The initial installation defaults to local as the storage area. This exists on your system in /data/storage/local. When you create new clusters, you can configure their storage areas in a different subdirectory, or different mount point altogether. It is designed to be flexible and map to most storage scenarios. This is termed the root storage path.

Layout and Structure

Below is a diagram of the default layout for storage.

Storage-layout.png

The /data tree contains self-explanatory subdirectories. Under /data/storage is where it begins to get interesting.

Data Splaying

With thousands of virtual instances, you don't want their storage areas all homed in the same directory. Firstly, you'll run out of inodes eventually. Secondly, as the more and more entries are added to the file system, access slows down considerably. To combat this performance decay, DreamFactory Enterprise builds a multi-layered storage structure using user information. The instance storage tree then becomes a simplistic hashed directory tree. We call this ***Data Splaying***.

Each Dashboard user is assigned a unique storage ID in the form of a GUID (User.storage_id_text). This GUID is hashed, and with it and the first two bytes are used to create a two-level nested subdirectory. Example with a fictitious user #1:

  • User id #1 has a storage ID of 00933DBC-C7A9-DE0E-5D38-FB23B574A580.
  • Hashing the storage ID gives us 9414f5052eead73c262d5ad89922541426208763ce86c2b64cc6a6680914a915.
  • From the hash, the first two bytes become the first directory level /94.
  • The hash itself is the final component /94/9414f5052eead73c262d5ad89922541426208763ce86c2b64cc6a6680914a915.


This final result is the relative path under the root storage path (which, as stated above, defaults to /data/storage/local). Therefore, the full path to the storage area of user id #1 is /data/storage/local/94/14/9414f5052eead73c262d5ad89922541426208763ce86c2b64cc6a6680914a915. This is termed a user's storage root path as all instances for this user will reside in this directory. Also in this directory is a private area that survives instance termination. Currently, nothing is stored in this directory.

Storage Utilities

The DreamFactory Enterprise storage support library provides a service called InstanceStorage. It can be used to retrieve instance storage information. A facade is also provided for easy access.

Usage

Adding the storage library to your own applications is simple. Just add the requirement "dreamfactory/dfe-storage": "1.0.*" to your composer.json file and run composer update. Once installed, add the provider to your config/app.php:

return [
    ...
    'providers' =&gt; [
    ...
        'DreamFactory\Enterprise\Storage\Providers\InstanceStorageServiceProvider::class,
    ...
    ],
    ...
];

Available Methods

The InstanceStorage facade offers the following useful methods for retrieving instance storage information. These come in two flavors. The first returns the path as a string. The second type returns a Flysystem object which represents the mount point. The $tag parameter is used to identify specific directories in the local cache.

Path Getters

  • static string getStoragePath(Instance $instance, $append = null, $create = false)
  • static string getPrivatePath(Instance $instance, $append = null, $create = false)
  • static string getOwnerPrivatePath(Instance $instance, $append = null, $create = false)
  • static string getSnapshotPath(Instance $instance, $append = null, $create = false)

Mount Getters

  • static Filesystem getStorageMount(Instance $instance, $tag = 'storage:instance-id')
  • static Filesystem getPrivateStorageMount(Instance $instance, $tag = 'private:instance-id')
  • static Filesystem getOwnerPrivateStorageMount(Instance $instance, $tag = 'owner-private:instance-id')
  • static Filesystem getSnapshotMount(Instance $instance, $tag = 'snapshot:instance-id')