PBMS Version 2 released

Version 2 of the PBMS daemon is now ready.

Here are the major changes introduced with this version:

• PBMS is fully integrated with MySQL 5.5:
  PBMS is now provided as a patch for MySQL 5.5 which simplifies installation and provides numerous benefits.
  
  
• All engines are "PBMS enabled":
  PBMS no longer requires that you have a "PBMS enabled" storage engine to be able to use PBMS.
  
  
• The MySQL client lib provides the PBMS client API:
  You no longer need to link your application to a separate PBMS lib to use the PBMS 'C' API.
  
  
• mysqldump understands PBMS BLOB URLS:
  When dumping tables or databases containing PBMS BLOB URLs mysqldump will dump the referenced BLOBs as binary data to a separate file. Since the BLOBs are dumped to their own file there is no need to convert them to hex data so they consumes only half the disk space they would have otherwise. The dump process is faster and uses less memory because the BLOBs are streamed directly from the PBMS daemon into the file.
  
  
• The mysql client handles the PBMS BLOB dump file:
  When restoring a database or table from a dump, the file into which the BLOB data was dumped can be passed as a command-line argument. The mysql client will then stream the BLOB data directly to the PBMS daemon which is faster and requires far less memory than if it where sent back to the server via 'insert' statements.
  
  
• A PBMS daemon ID is part of the BLOB URL:
  Each PBMS daemon has its own unique ID number. This allows the PBMS daemon to recognize and handle inserts of PBMS BLOB URLs from other PBMS daemons. A PBMS system table is provided into which daemon information can be inserted for other remote PBMS daemons.
  
  
• BLOB replication is handled automatically:
  When a PBMS BLOB URL is inserted int a table on a slave server the PBMS daemon recognizes that the URL comes from another PBMS daemon and so it sends a request to the original daemon and pulls the BLOB across so that it is now replicated locally.
  
  
• New internal BLOB indexing system:
  A new PBMS BLOB indexing system improves BLOB access performance and BLOB tracking.
  
  
• PBMS system tables are now indexed:
  The PBMS system tables that provide access to BLOB metadata are now indexed so that accessing the tables no longer automatically results in a table scan of the entire BLOB repository.

This is a major version change and as a result is not backward compatible with the earlier versions of PBMS. If you have an older installation that you need to upgrade please contact me and I will give you details on how best to do this.

The documentation and web page for PBMS has also been updated so I invite to check it out at: http://www.blobstreaming.org

You may notice a change in the documentation with regards to the definition of PBMS. The abbreviation 'PBMS' is being kept but what it stands for is being changed.  Originally it stood for "PrimeBase Media Streaming" but it occurred to me that when a DBA is designing a database system they are not likely to ask them selves the question "How will I stream my media?" but they may well ask them selves "How will I manage my BLOBs?".  So PBMS now stands for "PrimeBase BLOB Management System" which I think is a little more intuitive.

New PBMS version

A new version of PBMS for drizzle has been pushed up to launchpad:

drizzle_pbmsV2

I have rewritten PBMS and changed the way that BLOBs are referenced in order to make PBMS more flexible and to fix some of it's limitations. I have also removed some of the more confusing parts of the code and reorganized it in an attempt to make it easier for people to find there way around it.

So apart form some cosmetic changes what is different?

Maybe the best answer would be to say what hasn't changed: the user and engine API  and the way in which the actual data is stored on the disk remains pretty much unchanged, but everything else has changed.

The best place to start is with the BLOB URL, the old URL looked like this:

"~*1261157929~5-128-6147b252-0-0-37"

the new URL looks like this:

"pbmsAdaVAQCAAAAAAAAAANaVAQCAAAAAAAAAAG30qzsGAAAAAAAAAAEAAACAAAAAAAAAAAEAAAAAAAAA"

which is obviously a lot more intuitive.  :)

OK maybe it is bigger and uglier but it contains a lot more information. It is actually a base64 URL encoding of a data  structure containing information about the BLOB that makes it universally locatable across different PBMS daemons running locally or remotely.

How is this done?

When a BLOB is uploaded to a PBMS daemon the URL generated for it contains, among other things, the PBMS daemon's server id as well as the database ID and the BLOB's repository index value. These 3 values remain with the BLOB for it's life regardless of what server or database it may eventually end up in.  This allows you to insert a BLOB URL from one database into another database, possibly on a different server, and the PBMS engine will be able to use the URL to look up the blob, if it cannot find it in the database's repository then PBMS will automatically fetch the BLOB from the source server or database.

When fetching the BLOB the current server id, database id and index, which are also stored in the URL, are used.

This means that the following will work:

insert into foo.blob_table1 select * from  bar.ablob_table;
insert into foo.blob_table2 select * from  bar.ablob_table;

The first insert will copy the BLOBs from the BLOB repository for database 'bar' into the repository for database 'foo'.

The second insert will will recognize that the BLOBs already exists in foo's BLOB repository and just add references to them.

The same would hold true if database 'foo' was on a different server on the other side of the world.

BLOBs and replication:

A practical use for this is with replication, the replication process replicates the BLOB URLS to the slave server and PBMS pulls the BLOB across automatically. You can try this out using drizzle replication and my drizzle_pbmsV2 branch.

The only thing you need to do is tell the slave server how to map the PBMS server ID to the actual server. You do that by inserting the information into the 'pbms_server' table in the slave machine's pbms database. The master server's PBMS server ID can be found by doing the following select on the master server:

select * from pbms.pbms_server;

Resulting in something like this:

+-------+-----------+------+--------------+
| Id    | Address   | Port | Description  |
+-------+-----------+------+--------------+
| 38358 | localhost | 8080 | This server. |
+-------+-----------+------+--------------+

Then on the slave server do the following insert:

 insert into pbms.pbms_server values(38358, "master_host", 8080, "The master replication server");

where "master_host" will be the same IP address as you have for "master-host" in the drizzle slave config file.

Note: The PBMS server ID is not the same as the drizzle server id.

What else:

With the new design of the PBMS daemon it would not be very difficult to create a stand alone BLOB repository server that could be used as a backup for BLOBs or a a central repository for a cluster of servers. 

The next step though is to update the PBMS documentation and build a version  for MySQL.