NFS to AWS S3 migration
My DevOps team decision to drop NFS took a while. The biggest problem was that we could not isolate issues with NFS server when we reached some level of application load. NFS was the main data storage for photos, videos, streams etc. The problem was with scaling this solution. It was central point of tens of our services and if it fails the production was basically stopped. Yeah, to risky to extend and stick to this solution forever if we experienced bugs which we can not isolate on source code level!
We had a few choices. Build own API service as file storage provider? or look for external provider. It is worth to mention that we used AWS CloudFront as CDN for a very long time. So finally the choice was AWS S3. It stores files, distribute content internally and handle backups. Also important aspect was that we wanna finally switch our bare metal solution to cloud only.
It was real challenge to refactor all our service to switch to S3 without any troubles and production downtime. There is always a lot of aspects to consider before you will start. Important is at least backup, restore, latency of network between data centers (application to AWS).
As result we had to:
- Upgrade CM Framework to operate on more abstract file system providers
- Introduce S3 (AWS SDK) layer for our framework
- Adjust Wowza (Media Streaming engine) services to work more closely to application (CM Framework)
- and many more (private repos)
Pleas have a look on our puppet implementation of:
Comments