Back to blog

Extraction Speed – It’s A Two Way Street

13 Mar 2015 by Orlaith Palmer

There has been a great deal of noise in the archive migration space around how speed of extraction should be the ultimate benchmark of an efficient migration project. Some say that to achieve optimal extraction speed depends on a direct connection to the storage system, bypassing the API of the vendor. But archive migration is a two-way street, extraction + ingestion, which is why it’s important to look both ways before you cross the road.

Don't Walk

In instances where you’re looking to migrate your entire data set from A to B in a single move, extraction speed may be high on your list of priorities but this will only be effective in a situation where your repository ingestion speed is as fast, or faster, than the source. In the majority of cases you need to look at your situation and understand if it’s really a good idea to extract faster than you can ingest.

Some approaches advocate a data remediation process whereby you ascertain the status of your archived data, such as duplicate or expired email items, before migration. This is more efficiently addressed by a smart filtering technology built into certain migration tools. Applications such as these can rapidly filter before extraction and would allow for a different workflow for orphaned archives.


Many of our customers have realised the benefits of migrating away from costly servers to cheaper, more extensible cloud storage in the shape of Office365. To do this effectively, and at speed, requires an intelligent, end-to-end management of the entire migration process, not just an ability to quickly pull data out of source systems.

This is where leveraging the API can prove its value. If you adopt an approach where the smaller items in an archive are identified and batched together before sending to Office365, you can send them in one go which means only one call to the API per item. Not only does this save you a great deal in network bandwidth, you can also make better use of a safer, vendor-approved connection, proven to better preserve the integrity of your data.

But what about ingestion? If Office365 feels like its being sent too much information, it will issue a back-off command to say it won’t accept any more content for the amount of time it needs to finish its task. This is where it’s crucial to remember that extraction speed is only half the story. Handling these commands intelligently, in order to optimise ingestion speed, is a critical factor in how long your project will take to complete, regardless of how quickly you’ve extracted your data.


In the case of back-off commands, it’s possible to understand how long it will take for the task to clean per mailbox, so you can hold off for that amount of time. By not sending anything to that mailbox during this period, you can free up ingestion potential for other mailboxes that may be available.

When considering a migration project, remember that it’s an end-to-end process that requires every element to be moving in optimal synchronicity to be truly efficient, much like any factory production line. You may have the best suppliers in the world, but without the processes in place to convert them into consumer goods, the capability is largely redundant. Look to industry-certified providers of migration tools to ensure your emails arrive safely and efficiently at their new source just “as nature intended.”

Follow us on L