Some background.
We nicknamed our data storage system TROVE. It's a tortured acronym of The Repository Of Vast Expanse. Each TROVE server has a RAID5 array of 5x1TB hard disks. This results in 4TB of usable storage per TROVE server in their current configuration. The TROVE servers are entry-level'ish servers that we built ourselves - entry level motherboard with plenty of USB ports, quad core CPU, 4GB of RAM, onboard gigabit ethernet, decent ATX PSU and tower ATX case. (Larger rack servers would have a much higher storage density than our current tower servers but they would be considerably more expensive.) The TROVE servers run Windows Server 2008.
So right now we have 4TB of protected storage per server. Want another 8TB of storage? Throw another two TROVEs into the mix. Ultimately we will migrate all of our storage to higher density solutions - still based on our TROVE concept - but for the time being it's 4TB/server.
Each TROVE shares its storage volume on the network for other computers to access.
FAT.
There are a number of places in the Mugurdy web application and supporting service applications that we read or write to the file system. As we don't have a single/contiguous large volume to write to (because we have multiple shares that we can store data into) we need something to act as a FAT (File Allocation Table) so that we can find the appropriate share to read/write to.
So we built a Microsoft SQL Server database under Microsoft SQL Server 2008 to act as our FAT. A simplified version of the database tables is as follows:
Each Filename in use throughout the system is based on a GUID, so it's guaranteed to be unique. To find where a filename is on the network we can then use the following stored procedure:
CREATE PROCEDURE [dbo].[GetFullFilePathFromName]
(
@filename nvarchar(255)
)
AS
SELECT
FileAllocationTable.Filename, FoldersAllocationTable.FolderName, ServersAllocationTable.ServerName
FROM
FileAllocationTable INNER JOIN FoldersAllocationTable ON FileAllocationTable.FolderID = FoldersAllocationTable.FolderID
INNER JOIN ServersAllocationTable ON FoldersAllocationTable.ServerID = ServersAllocationTable.ServerID
WHERE
FileAllocationTable.Filename = @FileName
As file locations rarely (if ever) change across the system, it's safe to cache the output from this stored procedure wherever it's being used. Each TROVE server runs a Window Service that updates the database with the amount of free disk space available on that local RAID volume. We update disk usage about once a minute.
When we want to write a file to the system we pick a random folder (that's available for writing), write the file to the file system, and if the write was successful, we then simply add a record to the FileAllocationTable with the appropriate FolderID. If the write was not successful we pick another random folder on the network, and repeat until we get a successful write. (We haven't seen any write fails yet, but we're already covered in case this does happen in the future). As a TROVE server starts to fill up we add another TROVE server onto the network. Data will then be written to that new TROVE server, and to any existing TROVEs (that are available for data writes). Once a TROVE goes below 20% storage capacity it is disabled for writing. It will still be enabled for data reads, but data writes are disabled.
Reading files.
Here is a simplified version of reading in an image file from a hard disk:
Dim FileName As String = "SomeImage.png"
'Assume the file is in c:\
Dim MyBitmap1 = New System.Drawing.Bitmap("c:\" & FileName)
And to read the same file from the TROVE system:
Dim FullPath As String = (New Foundation.FileStorage).GetFullFilePathFromName(FileName)
Dim MyBitmap2 = New System.Drawing.Bitmap(FullPath)
So you can see it doesn't require very much code, and only requires very minimal changes to application logic to get it working.
Looking to the future.
We're working on adding another level of redundancy to the system. At present the entire system can maintain a single drive failure per TROVE server and maintain normal operation. However, if two drives failed in a single server, or the server itself failed, then all of the files stored on that server will be offline.
Soon files will be located on multiple servers across the network. This will mean that the system will be able to maintain normal operation from multiple file server failures across the network.
Subscribe