Sometimes I mentor developers. I always tell them this one thing they should aim for: find something you love and build a project around it with the technologies you want to master. Practise what you preach, so that is exactly what I did myself! This is the story of how my pet project got out of hand and needed to be migrated to the cloud.
I love Magic. No, not the kind where you saw someone in half and put them back together, but the trading card game “Magic: the Gathering”. A lot of people know or are familiar with this game, but don’t realise there is a huge second-hand market surrounding it. You see, not every card is as common as the other, and some cards are more powerful within the game than others. Supply and demand, a tale (almost) as old as time. Just last week a single card was sold for over $500.000.
Anyway, I was in college and wanted to know more about this second-hand market for cards so that I could buy the cards I wanted at the right time. Not to speculate, but to play with. I wanted to fetch the prices daily and plot them in a chart on a website, so I would have an idea how these prices worked. My first iteration was done in good-old PHP. But after a while I wanted to learn something new and focussed on Ruby on Rails. So that’s what I did. I re-created everything in Ruby, learning along the way, and thus MTGStocks was born. The site was meant for me and my friends to keep track of prices of the cards, but it “leaked” and I noticed an uptick in users.
|
When Angular 2 RC.5 came out I wanted to learn how to use it. So I stripped the Ruby on Rails backend to function only as REST API, and I created a new frontend. Again, learning along the way, because doing is learning. In the meantime the number of users actually using the website just kept creeping up. Up until about 6 months ago the entire website was running on a single Virtual Private Server (VPS), serving millions of pageviews.
Risks of my legacy VPS
You can probably see where this is going, as this single VPS had a few issues and risks:
- I had to maintain it myself and make sure it kept up with security updates.
- Resources were limited. The VPS was, in the end, constantly running at full memory and/or CPU and crashing occasionally because of it.
- Single point of failure. If the VPS crashed, the entire website would be down, and my users would lose all benefits of the website.
- As I gathered more historical data, I came to a point where I want to start running more analysis on it, and with this single VPS I simply did not have the resources anymore.
Also with eventual expansions and new features this no longer was a maintainable situation. I needed to migrate! And thus arose a new opportunity to learn something. I could have gotten more VPSs and spread the load that way. But that wouldn’t fix risk #1; it would even make that a larger risk with more work involved. Since I was already familiar with AWS, I decided to move there.
Properly managing my cloud infrastructure
I also decided I needed to be able to replicate my environments. Not only did I want a production infrastructure, but also at least one for development and testing. I used Terraform to define it as Infrastructure as Code (IaC). Anyone who wants to do something (semi-)serious in the cloud should use IaC. It provides you with:
- Speed and simplicity: spin up an entire (extra) infrastructure by running a script
- Configuration consistency: avoid human error by making sure each change in the infrastructure only changes what you intend to change and make sure all environments are the “same”
- Cost efficiency: you don’t have to spend extra time on manually changing your infrastructure and focus your time on development
- Risk management: automated processes have less risk of human error. When you keep your IaC in a repository you can also keep track of its history and changes
The actual migration
Since I feel really comfortable in Angular and frontend, I decided to migrate the frontend first. This seemed like a good place to start. I did my research, put it all in Terraform, made sure I could spin up a development and production environment, and gave it a try. This led to putting the frontend in an AWS S3 bucket with a Cloudfront as CDN to cache the assets. A Codepipeline could fetch my code from the repository, build it, and store it in the bucket.
Although this was fun to do, and something that needed to be done, it barely took any load of from the poor VPS behind the curtains. Therefore I migrated the backend to AWS Lambdas and put them in an API Gateway. Since most of the requests could be cached for several hours, I also put a Cloudfront in front of the gateway.
My biggest challenge was moving the database. I had to move tens of gigabytes of data from the old server to AWS RDS. I chose for an Aurora RDS because it is more fault-tolerant, has higher performance, and is more consistent than the non-Aurora counterpart. After a few test-runs I felt comfortable enough to actually make the switch.
Reaping the benefits
Of course it took a bit more effort and time than the above paragraphs make it look, but the benefits for me are obvious.
- Performance: Because of auto-scaling of AWS Lambdas and RDS it makes sure I have extra resources when I need it. Cloudfront makes sure common assets are cached around the globe serving the users faster than the site ever did before.
- Costs: Because of the auto-scaling mentioned above, I don’t have to pay for a huge server for a few peak-moments a week. I only pay for what I use and need at that particular time.
- Redundancy: No longer a single point of failure. The website is always up, and I don’t have to worry about servers crashing while I’m asleep.
- Future proof: Knowing that I can scale my resources to my needs, I can actually start planning the next steps. My infrastructure is no longer the bottleneck for advancement.
- Time savings: I don’t have to spend time in updating, maintaining, and monitoring the health of my hardware.
Not only am I now running fully in the cloud, but it has made me more aware of processes. In the cloud you pay as you go and what you use, meaning I am more focussed on whether my processes are as efficient as they can be. This is not only leading to a reduction in costs, but also in performance for my users.
Who am I?
My name is Arjen, I’m a senior fullstack engineer and consultant working as Team Lead for Techspire in the Netherlands 🇳🇱. I have been working in software engineering for almost 10 years and still love it every single day. My main expertise lies with Angular. As a true geek I love to expand my knowledge and am proficient with, amongst others, Python, Ruby on Rails and AWS.
Do you think you have what it takes to work with us? At Techspire we’re looking for people who love technology as much as we do, and are looking to push themselves to expand their knowledge. Also, we love a good story, a good laugh, and a few beers.
See all Techspire blogs?
See all Techspire news?
Follow our LinkedIn page: Techspire LinkedIn
