Moved to self-hosting with Coolify

Reason

There are plenty of horror stories with obscene bills from cloud providers. Fortunatelly, I am not in that category, but I don’t want to become complacent either.

The move to self-hosting has to be like pulling a band-aid, fast and the sooner the better. When you get locked in all of the juicy perks the vendors have to offer, it’s very hard to live without them.

Old infrastructure

GetSite started with a hybrid infrastructure. Some services were deployed on Hetzner with Rancher, and others were split between AWS and MongoDB.

On the MongoDB cloud, I was using Atlas and ran a serverless instance for the database. The invoices started to ramp up, but not in direct proportion with the traffic we were getting. At first it was a couple of bucks per month and very quickly got up to $40 / mo. As you might have guessed, this was a strong motivator to move from Mongo.

Before making the move, I tried to figure out what inflated the invoices so much. The database instance was getting 20 req/s at all times, and had no idea why. The official support was quite unresponsive and not helpful in this regard.

Mind you, this spike in requests has started about a month ago, however the traffic to GetSite didn’t increase. Something’s fishy if you ask me, but I didn’t want to spend more time looking into it.

On AWS, I had few Elastic Beanstalk instances that were quite expensive in comparison with Hetzner. The instances were part of an autoscaling group.

Did I mention I was using Rancher 1 for the self hosted services? It is a great piece of software, but it was discontinued a while back. Rancher 2 is very complex to use and geared towards enterprise users and big clusters.

New infrastructure

Enter Coolify, the successor to Rancher. I have been trying to find a replacement for Rancher for a couple of years now. It seems like all container managers (Portainer included) are now tailored to the enterprise. I read about Coolify a few months ago, but wasn’t convinced. Their website is not doing them justice.

A couple of weeks ago, after the inflated MongoDB invoice, I decided to jump in and install Coolify on a Hetzner server. The onboarding process is a bit unintuitive, but it grew on me as I started using it.

I moved all services from AWS, MongoDB, Rancher, and some old MySQL and Redis servers to Coolify in a day. Spent another day setting up workers, load balancers, backups, security and testing everything out. All of this while keeping 100% uptime (ok… maybe like 99.99%).

Costs

I cut costs down to about $40 / month for the whole infrastructure. That’s how much the last MongoDB invoice was, for reference. While bringing the cost down, I also beefed up the server capacity. The current setup has plenty of spare room for future growth.

Conclusion

This isn’t a free lunch. I had the skills to build my own infrastructure and configure everything in a way that doesn’t keep me up at night. If you don’t have some experience with DevOps, it might get tricky.

There are things I miss like autoscaling, integrated services like charts and analytics from MongoDB, but I can live without them. Moreover, I can get to install Grafana.