Back to blog

The final morsel of cheese

3 Oct 2018 by Jason Jacobo

Part 4: Keeping it light with Cargo Bay

Migrations are a messy business. This has always been true. It doesn’t matter if you are using a migration solution or native tools to move the data. Systems were just never designed to be migrated in any short duration. Because of this, and the needs of the customer’s performing them, migrations frequently require the ability to log and track what transpired during the movement of this data. Most (if not all) native solutions are unable to do this. In a sizable migration, tracking these items can result in many millions of records related to source items …even billions in some instances.

Missed all the other ‘morsels’? Click the links here for part 1, 2, and 3 of the series.

As discussed in prior postings (see above, if you missed them), this very fact poses a problem for migration solutions. How can you track the progress of all the items in a source system, as well as what happens to them during a large migration project, without encountering performance degradation due to database table bloat? Frequently, the same tables in a database used to track these items are also used to make the migration happen. This means that every action of a solution could be impacted by the size and contents of this table, resulting in slower performance the more items that need to be migrated.

Although creating indices and refining queries may help improve retrieval performance, it can negatively impact the speed when loading a database with data. Migration solutions like Archive Shuttle may load data into a system 70% of the time, minimizing the value of these approaches and not addressing the root cause of the issue.

So, what should you do with this data?

This data needs to be stored because you may want to review it again but does not really need to be persistently or currently accessed. Think of it like a collection of photographs you’ve taken recently. You may enjoy looking at them shortly after you took them, but would you want to keep them on your desk for the years to come? This storage method could cause a lot of clutter! More than likely, you want to preserve these photos. You would want to move those images from your present desktop to a storage container, like an album or something. Somewhere they’ll be safe and not subject to the daily activities taking place on your desk. As it happens, Space Shuttles also have a safe, designated storage location…

Enter Cargo Bay

In Archive Shuttle 9.0, Quadrotech brings to its customers Cargo Bay, a feature designed to finally address the issue of table bloat when performing large email archive migrations. Cargo Bay leverages Archive Shuttles’ modular database design to create a database that can be leveraged to retain the chain of custody for your migration, outside of the active tables used to perform regular product operations.

Saving space and time

Many customers have a requirement or desire to retain the granular details of what transpired in their migration, but only for reporting after data processing is completed. By removing completed mappings from the active tables, we effectively free up those tables to reduce the size and burden of querying against them. The net result is that queries required for the processing of data run fast, and complete quickly, improving the performance of any functions of the solutions that leverage this data. Examples of operations that leverage this data would be filtering, item collection, the UI, and (perhaps most importantly) routing for exporting and importing.

With the introduction of Cargo Bay, Archive Shuttle now has two ways to support even the most massive migrations in the field. For those having no evidentiary need for migration chain of custody, we can configure the solution not to retain it. However, if this information is important to you or your organization, Archive Shuttle can still retain it without impacting the performance of your migration as it progresses, and the environment grows.

What does this look like in practice?

After its initial release, we put Cargo Bay to the test. An average-sized production customer who had migrated over 35million items with our hosted solution was utilized to grab some performance metrics. Our baseline queries (prior to archiving any of the mappings of this migration) took over 40 minutes to run to completion.

The following table represents the improvement we noticed in applicable query performance after using Cargo Bay against this Archive Shuttle instance:

Results archived Query execution improvement
5% 20%
36% 50%
58% 71%
90% 97%


This means that a query that took over 40 minutes to execute with no items archived, took less than 20 minutes after slightly over a third of the content was archived into Cargo Bay. All consecutive queries against that table took half of the time to complete. The process to archive this data took less than 3 minutes to complete, and every related action after this data was archived was performed more promptly due to a smaller table size.

Cargo Bay was built to address the needs of the largest organizations. The businesses with vast or complex email eco-systems, who have been using enterprise archiving solutions for data management, compliance, risk mitigation, or legal discovery. These organizations require a migration solution built to scale to their needs, their environment, and their security. Cargo Bay adds to the arsenal of functionality built into the underlying core of Archive Shuttle that makes it the clear choice for a centrally managed single environment to perform the largest migrations in the market.

To learn more about Cargo Bay, Archive Shuttle, or any other Quadrotech solution, please feel free to reach out to us.