Back to blog

See why Archive Shuttle should be your #1 choice for migrating email archives to Office 365

3 Dec 2014 by Orlaith Palmer

Migrating Archives At Scale

We are often asked during evaluations whether our products are truly scalable, or whether we are suitable for smaller migrations. Well, the answer to both questions is an unequivocal YES.

Archive Shuttle has been used to migrate archive emails to Office 365 in excess of 250 TB  so our products are just as usable for a 5 TB migration. Indeed, because we allow cloud-controlled migration, where a migration server is not required on-site, several of our customers are using us to migrate 2 TB archives quickly and at a realistic cost.

In cooperation with Delegate, a premium European Microsoft and Quadrotech partner based in Denmark, and a customer with three different EV sites in three different continents, we tested our Advanced Ingestion Protocol into Office 365 and compared it to the traditional Exchange Web Service Approach (EWS).

Today we received the first “real-life” results:

The data was migrated during the POC three different times using two different approaches. In the first approach, pure EWS (Exchange Web Services) was used, resulting in a peak performance of 11 items/second in EMEA, 11 items/second in USA and 9 items/second in APAC. The average performance was around 24 items/second.

In the second attempt, a fully functional AIP Office365 Module in Archive Shuttle was used to assess the real-world performance increase when ingesting the items into Office365.

The performance increase was significant, resulting in a peak performance of 55 items/s in EMEA, 48 items/s in USA and 44 items/s in APAC, with each region using just 8 ingestion threads. The average performance showed a performance of around 125 items/second across all three regions.

Based on the performance numbers above, the customers migration duration has dropped significantly. The effect is shown in the estimates below:

Archived Items Count in EV: 520,817,280

Approach 1: (traditional EWS)
520,817,280 / 24 (items/s) / 60 (sec) / 60 (min) / 20 (h/day) = 336 days (net) migration time

Approach 2: (Advanced Ingestion Protocol)
520,817,280 / 125 (items/s) / 60 (sec) / 60 (min) / 20 (h/day) = 58 days (net) days migration time

The Quadrotech Research Team are always working on improving project efficiency and migration speeds to help customers complete migrations as fast as possible.

The Advanced Ingestion Protocol will be released with Archive Shuttle 6.6. As our research team optimizes our technology further, you can expect more exciting Office 365 innovation announcements very soon!

Selective Migration

In many instances, it will be desirable to limit the amount of data that is migrated to the new environment.

From a business perspective it is essential to follow email retention policies, and these are likely to include deletion of emails that are older than a certain time, typically 7 years.

From a technical point of view filtering out message classes (for example Notes, Tasks, Drafts etc.), shortcuts and other unwanted data will be required.

Archive Shuttle includes a powerful filtering engine that caters for powerful filter stacks on multiple email attributes including:

– Item date
– Archived Date
– Retention Category
– Folder in Archive
– Shortcut status
– Message Class

As a result the new target environment can be guaranteed to only include data that is current and relevant.

Coordinated Peer-to-Peer Architecture

Archive Shuttle employs a coordinated peer-to-peer architecture that is designed for maximum scalability and resiliency.

The coordination is carried out by a CORE server which passes instructions to peer-level agents that are responsible for extraction, ingestion, synchronization, switching etc. This approach ensures central coordination, control and reporting while providing a totally scalable solution with multiple agents carrying out the heavy lifting.

ArchiveShuttle - Architecture
In the simplified example above, which represents the migration of a single archive from Enterprise Vault to Microsoft Exchange 2013, a workflow will be executed as follows:1) The CORE will identify an available source agent and send an export command detailing the export criteria
2) The source agent will export data to a staging area as instructed
3) Upon successful completion the source agent will send a notification to the CORE
4) The CORE will identify an available target agent and send an import command detailing the import criteria
5) The target agent will import the data directly from the staging area
6) Upon successful completion the target agent will notify the core

At this point the data has been successfully synchronized, the CORE will now issue the necessary commands to source and target agents to initiate the switch, so that the “Sync’n’Switch” is complete.

This architecture is well suited to on-premise and hybrid cloud deployment where the CORE can be located remotely. It also lends itself well to very fast performance using a multi-threaded approach. Real world migration speeds for a  migration of Enterprise Vault from Centera to Netapp are shown below. In this case three export and three import agents were used under the control of a single CORE.

ArchiveShuttle - Extraction Speed

 

Multiple Sources and Targets

For maximum flexibility Archive Shuttle supports migration between a multitude of archive sources and archive or mailbox targets. The include:

Sources
  • Symantec Enterprise Vault
  • The HP archive family including Zantaz and Autonomy
  • EMC Source One
  • HP Autonomy / Zantaz / EAS
  • EMC EmailXtender
Targets
  • Symantec Enterprise Vault
  • Microsoft Exchange
  • Microsoft Office 365
  • Mimecast
  • Proofpoint
  • Global Relay

In addition Archive Shuttle can import from PST Files, .eml and .msg files allowing even more flexibility for migration.

The modular nature of Archive Shuttle means that each source and target is optimized according to their own requirements.

Each target system is unique and treated differently for the best possible experience. As an example, Quadrotech has implemented functionality to take advantage of the latest Office 365 “capabilities” and is working closely with Microsoft to implement advanced migration technology into Office 365 and the Quadrotech Products.

Another example is the Exchange 0n-premise ingestion module where we utilize two completely different protocols (MAPI & EWS) to make sure that the customer experiences a smooth migration.

A third example is the Enterprise Vault ingestion module, where it is the fastest in the market. Years of optimization and close work with the Symantec Engineering teams on the APIs allow Archive Shuttle to adopt a massively multi-threaded approach with stunning migration rates.

Can archives be merged?

Yes. If you have two different Enterprise Vault sites, you can use Archive Shuttle to merge those two sites into a third one or you can simply export a complete Enterprise Vault installation and integrate it into an existing site.