This is the second installment of a multi-part series. You can start at the beginning with Building an RPM repository in AWS.
What just happened? Turns out you were relying on a repository that pruned really old packages. And they just pruned one that your application relies on. So what are you going to do? Try like hell to find it somewhere, anywhere, please-God-let-me-be-able-to-find-it. Good luck with that.
Now, we’re going to talk about how you can prevent this occurrence in the future.
Let’s outline our guiding principles, and then describe an architecture that will help us make those principles real.
- I will host locally all packages that my infrastructure and application rely on. I will not rely on someone else to keep my packages available.
- I want to support having more than one repository server, all with the exact same packages on them. (This might be useful, for instance, if you have some web servers in the US and others in the UK. In this case, you can have a repository server in the US and an identical one in the UK.)
- I want to be able to publish new packages. This includes updated custom packages as well as updated base packages.
- My repository servers will be built according to the same fundamental principles as all other servers in my infrastructure. This means using CloudFormation, Chef, etc.
With those in mind, let’s take a look at a high-level architecture that will deliver a usable solution.
Let’s walk through this.
- First, you’re going to need to download all the packages from a remote repository. (Think rsync.) They’ll live (only temporarily) on a development server.
- Second, let’s upload all those packages to an S3 bucket. Why there? First, I like the storage reliability that S3 provides. Second, when have updated packages, we’ll already have a place to put them. We could keep all of our packages in EBS, but I’d be afraid of accidentally deleting the volume. To increase durability, we could backup the EBS volume into a new image. But, then, each time we updated our packages, we’d have to make a new image.
- Third, let’s update our CloudFormation stacks and templates (not shown in diagram) to build out our new repo servers. As part of the build, we should download the files from S3 and create the repository.
- Fourth, we’ll modify our Chef scripts to tell our other servers to use the new, local repository server.
In the next installment, we’ll dive into the details and share some scripts to help make all this possible. To be continued.