Speed up migrations to Office 365 by reducing WAN data volume

The Highly Optimized Transport System (HOTS) can radically speed up migrations to Office 365 by reducing the volume of WAN data that needs to be transferred, and further optimizing performance over that of AIP. Tests indicate that this improvement is as much as a further 55 percent.

HOTS is a proprietary format that helps overcome the bandwidth limitations, single-mailbox restriction and performance throttling protocols enforced by or absent from the Microsoft Office 365 APIs. By combining the power of the Microsoft Azure cloud, our own unique bridgehead technology, and streaming optimizations, we ingest data in a way never seen before with Quadrotech’s Archive Shuttle.cloud (AS.cloud).

Whilst AIP and AIPv2 as protocols significantly reduce the bandwidth required through compression and optimization of data, and significantly increase performance by limiting the impact of throttling, HOTS further pushes the boundaries of what is possible.

Current migration practice typically uses migration products such as Archive Shuttle to transform extracted files into formats that push directly into Office 365, such as Quadrotech’s Advanced Ingestion Protocol (AIP), that are recognized by Exchange.

HOTS creates a WAN-optimized payload that shrinks data volume while keeping it intact and compliant. This payload is transmitted straight into Azure, where it is converted on the fly into AIP format for ingestion by Office 365. The process is orchestrated by Archive Shuttle modules running on both the customer’s on premises legacy server and Azure, meaning the main payload does not need to touch the AS.cloud server.

Begin the ingestion process faster

Migrating the email ecosystem to Office 365 efficiently can be a challenge for organizations, but HOTS revolutionizes the entire process by creating a way for Microsoft’s two clouds (Azure and Office 365) do all the heavy lifting, with Archive Shuttle.cloud providing the intelligence.

In a typical example where a customer has 5,000 Exchange mailboxes, archived every 30 days into something like Enterprise Vault (EV), there would perhaps be 10 TB of archives. The customer would naturally prefer to use WAN rather than disk for its Office 365 migration, but the sheer size of data would make this a slow process. The de facto standard for getting data into Exchange Online is EWS (Exchange Web Services) but, because of the way its coding works, the actual data that needs to be transferred more than doubles. Even with MSG or PST formats the size increases.

Tests using an independent tool that measures network volume indicate that 10 TB of data would create WAN traffic of about 12 TB (MSG), 15 TB (PST) or 22 TB (EWS). By comparison, HOTS can shrink the same traffic to nearer 9 TB – a massive 55 percent reduction compared with EWS.

Comparison with existing migration process

The steps commonly taken to migrate an item from a source system to Office 365 are:

Extract the item

The migration application connects to the source system and extracts an item. In most email archiving systems, the application typically results in a MSG or an EML File (or stream) containing the migration payload.

Store the item

The extracted item needs to be stored after extraction. Depending on the migration product used, it might be stored in the memory of the migration server or, in the case of Archive Shuttle’s modular architecture, in the staging area on disk to be picked up from the ingestion module.

Transform the item

Exchange does not understand the native MSG format used by Microsoft Outlook. Depending on the ingestion protocol one would therefore need to add the item either into a PST File (for PST import service), to transform it into an XML intermediate format (for EWS), or create a compressed sequence-reduced packet (for AIP).

Transfer and ingest the item

After the transformation has happened, a connection to Exchange Online is established and the item is transferred over the WAN to the Microsoft’s Office 365 datacenter for ingestion.

As explained above, it is very important that the transformation of the payload takes place to ensure that Exchange Online gets the data in a format that it can understand and process. However, the formats/protocols that Exchange currently understands are not optimized for WAN transfer.

HOTS reorders these process steps, combines them with Archive Shuttle’s modular architecture, and creates a proprietary, highly WAN transfer-optimized format that in parallel ensures also the compliance of the payload when migrating to Office 365.

No hardware for migration purposes is required on premises. AS.cloud controls the Archive Shuttle Modules, which are installed on the archiving (on premises) server. Data extracted from the legacy archive is automatically transformed into the WAN transport optimized HOTS format, shrinking the data volume while keeping the migration payload compliant and intact.

Archive Shuttle’s semi-synchronous migration approach allows an Archive Shuttle Bridgehead to be installed in Microsoft Azure (only one VM – A3 or A4 required). This accesses the HOTS data and converts it on the fly to an AIP blob that can be ingested directly into the Exchange Online mailbox or archive in Office 365.