Etsy Completes Five-Year Migration of Its 1,000-Shard, 425 TB MySQL Architecture to Vitess
Etsy has finished migrating its proprietary 1,000-shard MySQL infrastructure to Vitess after five years, 2,500 pull requests, and 6,000 query rewrites, eliminating a single point of failure while preserving its existing data layout through custom vindexes.
Overview
Etsy has completed a migration that took five years, roughly 2,500 pull requests, and the rewriting of approximately 6,000 database queries: moving shard routing for its entire production MySQL estate from a proprietary in-house system to Vitess, the open-source database clustering project that graduated from the Cloud Native Computing Foundation in 2019. The e-commerce marketplace detailed the effort in a Code as Craft engineering blog post published in March and covered by InfoQ on April 11.
The migration touched approximately 1,000 shards holding 425 terabytes of data and serving 1.7 million requests per second, making it one of the largest publicly documented Vitess adoptions to date.
What We Know
The Legacy Architecture
Etsy has relied on a sharded MySQL architecture since roughly 2010. The system used over 30 different ID types as sharding keys, primarily shop_id and user_id, with a proprietary ORM that treated each shard as a separate database and queried specific shards directly. A critical component was a single unsharded “index” database that stored the mapping between records and their corresponding shards.
According to InfoQ’s reporting, this index database represented a single point of failure. If it went down, the ORM could no longer route queries to any shard, risking a full site outage. The architecture also made it difficult to reshard data or shard previously unsharded tables without extensive manual effort.
Why Vitess
Vitess is a CNCF graduated project originally built at YouTube in 2010 to solve MySQL scaling problems. It provides horizontal sharding, connection pooling, and automated failover while preserving MySQL compatibility. Companies including GitHub, Slack, Square, Pinterest, and Shopify run production workloads on Vitess, according to the CNCF project page.
For Etsy, the appeal was threefold: eliminating the index database as a single point of failure, transferring shard routing complexity from application code into infrastructure, and gaining the ability to reshard data and shard previously unsharded tables.
The Custom Vindex Approach
The most significant engineering challenge was that Etsy’s existing shard mappings were random rather than algorithmic. Using Vitess’s out-of-the-box vindexes, which define how data maps to shards and route queries accordingly, would have required re-sharding all existing data. Senior software engineer Ella Yarmo-Gray told InfoQ that such a process “would be manual and likely take years.”
Instead, the team built two custom vindexes. The first was a SQLite lookup vindex that reads shard information from a SQLite database file. Etsy chose SQLite because it provided low-latency reads and a footprint small enough to copy directly onto each Vitess server, avoiding the latency and dependency of an external database call.
The second was a hybrid vindex that applies one of two routing strategies based on a threshold value. Records with IDs below the threshold use the SQLite lookup to find their shard, preserving the legacy mapping. Records with IDs above the threshold use a hash-based vindex for algorithmic routing. This approach allowed Etsy to introduce Vitess vindexes without physically moving any existing data.
Gradual Traffic Migration
Etsy’s experimentation framework enabled the team to gradually ramp up traffic through Vitess vindexes. For each table, they incrementally increased the percentage of queries using Vitess-based shard routing, allowing them to assess performance differences and catch regressions before committing fully. The transition moved shard routing from Etsy’s internal systems to Vitess, enabling capabilities such as resharding and the sharding of previously unsharded tables.
Yarmo-Gray acknowledged that the effort was “still a challenge to replace the database infrastructure for a codebase of Etsy’s scale and age,” according to InfoQ.
The Payments Migration
The shard routing migration followed an earlier, separate effort to move Etsy’s payments infrastructure to Vitess. Between December 2020 and May 2022, the Payments Platform, Database Reliability Engineering, and Data Access Platform teams migrated 23 tables totaling over 40 billion rows from four unsharded payments databases into a single sharded environment managed by Vitess. That project required adding a shop_id column to existing tables and backfilling values across billions of rows while the databases were resource-constrained.
What We Don’t Know
Etsy has not disclosed the full team size dedicated to the five-year migration, nor has it shared detailed performance benchmarks comparing the legacy routing system to Vitess-based routing. The company has not stated whether the migration resulted in measurable latency improvements or cost savings. It is also unclear whether any production incidents occurred during the gradual traffic cutover, or how Etsy plans to leverage the new resharding capabilities now that Vitess is in place.
Analysis
Etsy’s migration is notable for what it did not do. Rather than treating the move to Vitess as an opportunity to redesign its data architecture from scratch, the team chose a preservation strategy: port the existing shard logic into Vitess custom vindexes, introduce new algorithmic routing only for new data, and migrate traffic gradually. The approach minimized risk at the cost of speed, stretching what might have been a shorter project into a five-year effort.
The SQLite-as-a-vindex-backend pattern is particularly instructive. By embedding shard lookup data in a local file on each Vitess node, Etsy avoided introducing yet another network dependency into the query path, a concern that would have been especially acute given that the entire motivation for the migration was to eliminate a single point of failure.
The project also underscores the maturity of Vitess as a platform. A CNCF graduated project since 2019 with more than 130 contributors at the time of graduation, Vitess now runs at companies spanning e-commerce, social media, fintech, and developer tooling. Etsy’s successful migration of a legacy, decade-old sharding system adds a new reference point: Vitess can accommodate not just greenfield deployments but also complex brownfield environments with idiosyncratic data models.
For organizations running aging MySQL sharding infrastructure, the lesson is clear but demanding. Modern tooling like Vitess can absorb legacy complexity through extension points such as custom vindexes, but the migration itself requires sustained, multi-year commitment and deep coordination between platform, application, and reliability engineering teams.