So if ever Amazon can innovate and they're better than us, and they're cheaper, or we can secure better deals wherever makes sense, their job is to advocate us moving the data back." "Their job is to provide the best, most reliable and cost-efficient storage for Dropbox. The reason we call it Storage is because their job is not to do Magic Pocket. "Magic Pocket is this very famous system, and often people say 'what's the Magic Pocket team?' We don't have one, we have the Storage Team. So does this mean Dropbox has dropped the cloud, and is essentially an on-premises business now? Our usage patterns are much simpler, and we understand them, so we can. "Amazon and the cloud have to solve really broad problems - just imagine all the different usage patterns for S3. "Looking back, it really turned out to be a great investment for both our velocity and the business," Le said. "What we've seen in the last couple of years is that we tend to move more things on-prem than towards the cloud for our core storage production," Le said, but declined to share the exact percentage. but we also increased the number of drives per chassis so we really increased the density quite a bit."īy 2016, the company said that it had moved around 90 percent of its files over to on-prem, comprising four data centers. "Back when we started, we worked with four terabyte drives. "We replace our hardware every four years, but have at least a couple of new generations in those four years," Le said. The company helps design its own custom servers, cramming more and more storage into its data centers. Some of the big players still don't use SMR." "They need to build a new file system, etc. "This is one of the examples where we were able to work closely with hard drive companies and were able to move much faster than some other companies," Le said. SMD disks can be much denser by writing new tracks that overlap part of the previously written magnetic track, somewhat like overlapping roof shingles. This allowed us to gain a lot of efficiencies from avoiding the file system, but also move quite a bit faster."įor example, the company could adopt shingled magnetic recording (SMR) hard disk drives without waiting for drivers to support them. We moved from using a file system to just managing the drive directly - we literally open the drive as a block device and then we have our own formats. "We also significantly improved the architecture. "It's not only the language we changed," Le said. This involved a huge amount of software work - including switching from programming language Go to Rust mid-way through to reduce memory use - and getting deeply involved with the hardware to ensure that every ounce of possible storage was squeezed out of a rack. Over a two-and-a-half-year period, the company built its own massive on-premises platform, officially launching it in 2015. This article appeared in Issue 42 of the DCD>Magazine. This, Le said, has allowed for significant cost savings and more control - but is not something that most other companies could easily replicate. The result was Magic Pocket, one of the largest data migrations off the cloud in web history. "It was only a few years later, when we really believed we could tackle this problem better for our needs, that we even tried." "We used AWS S3 because storage at scale was an extremely hard problem to solve," Le said. It didn’t take long for the company to wonder whether it made more sense to do it themselves. The company has always had its own data center presence, but Dropbox needed more capacity and soon grew to become a major customer of Amazon S3 (Amazon Simple Storage Service) after joining in 2013. When file hosting service Dropbox first announced its hybrid cloud effort Magic Pocket in 2016, many saw it as a sign that the company was done with Amazon Web Services and was betting on an on-premise future.īut the reality is more nuanced, lead developer Preslav Le told DCD.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |