Ask HN: What design and architecture patterns have served you well?
There are tons of conflicting opinions on the best approach to architecting and designing large scale software. I'd love to hear some anecdotes for what's really worked in the real world.
Bonus points if the pattern was used in a large scale web app with multiple clients and a common backend api.
PHP front end, Redis for caching, proprietary third-party CMS for content/data entry and ETL, Cloudflare for CDN, home-grown user API written in Node.js with a MongoDB datastore. Mobile apps in their respective native languages. The user API and front end were deployed separately as EC2 instances in AWS auto-scaling groups. New Relic and Cloudwatch for monitoring.
Roughly 100 million page hits per month on the web app. Ran smooth as silk.
> PHP front end
Do you mean that you are fetching data from an internal API and sending server-rendered HTML response to the browser? I always naively thought GitHub worked this way. Build a GraphQL/REST API that exposes all of your system's internals, query/mutate through it and send an HTML response in the "frontend" (Ruby on Rails etc.). Frontend, but in terms of an application server that in turn accesses everything through an internal API.
Fetching and collating data from multiple other APIs, then creating server-rendered HTML responses to the browser.
Most of the Javascript was not for interaction, but for serving ads, really.
Interestingly enough, assuming you use React this is how I see the role of something like Remix[0], which serves as a "rendering layer" that aggregates application data from different API's/stores and then sends a cachable HTML representation to the client which is thereafter hydrated.
It's not quite a backend and it's not quite a frontend, but it sort of sits in the middle and (hopefully) helps you achieve lower latency. I haven't used it myself but I think it's a promising idea and based on your post I gather a similar concept worked very well for you.
0: https://remix.run/
What CMSs did you find to be the most effective?
How did they compare with open-source ones (Drupal et al)?
Do you find serverless CMSs to be more useful than traditional ones or vice versa?
Communicating Sequential Processes, and Actor model.
I know these aren't "design patterns" in the ordinary sense, but they seem to be ultimate models of computation. Every real world problem I've encountered has been much more easier to solve when looking at the system as a set of sequential processes that communicate.
Best of all - you can do it in almost every system out there. Shell + C is the oldest CSP "framework" I can think of.
Perforated Design
You shouldn't split your new system into a zillion pieces. But you should design it so it comes apart really easily.
I haven't implemented this pattern (and I'm not convinced it is widespread or even exists in a cohesive form yet) but I've become quite bedazzled with the idea of "UI as pure function of state" a la React/Redux, but with state maintained in the backend database and transmitted to the client via websockets, instead of locally.
Assuming a real-time interactive application, the major components would look something like this:
- Backend service distributed geographically to facilitate low latency to clients, with a single writer node and many read replicas
- Backend data store is SQLite, and importantly runs as a process on the same VM as each of the service nodes, so that there is no need for any network roundtrip to query the database on a read operation
- Clients subscribe to the service and receive a single JSON object representing the UI state
- Database modifications trigger a websocket notification to all subscribed clients with the respective diffs to the state object
- On the client side, a library like React (my choice, but of course there are others) watches the state object and renders the UI accordingly. Client writes are routed to the writer node (Traditional caveats about RYOW apply), UI updates are optimistically applied
- Pure client side state (modals, etc.) is handled solely via the frontend
It's tangential to the architecture, but this obviously would be well suited for something like fly.io. I'm sure many of the Phoenix people are nodding their heads as this architecture bears a lot of resemblance to LiveView, which I am also interested in for its own merits.
I see this as essentially a marriage between React/Redux and LiveView; it essentially comes down to having a globally replicated SQLite DB serve as your Redux store, but "over the wire" and it's colocated with your API layer.
What's the motivation? For me, React/Redux with Redux Toolkit is pure bliss for any sort of UI development. Throw in Tailwindcss and you're really rockin'. The problems arise from the impedance mismatch between maintaining application state locally in a Redux store when the true source of truth is your DB, and all the network traffic/caching considerations/api logic that needs to come with that. (You've actually got a distributed system on your hands)
If you can keep the client focused on the UI, then the complexity mostly lives in the API service layer where writes are replicated to all the VMs. It wouldn't work for every style of app, but with a service like fly.io behind you this tradeoff starts to look very attractive.
(Aside: I think this is why a lot of HN'ers are so excited about fly.io. If I can distribute my application and achieve super low latency relatively simply that is a step-change and opens up so many avenues of experimentation for development)
If the application were a more traditional read-heavy website and didn't need real-time updates, I might rip out the websocket layer and replace it with something like Remix, which I see as sort of a "rendering layer" that sits between the client and the backend for React apps.
Anyway, it's an exciting time to be a web dev. I would be curious to hear if anyone else has pondered a setup like this.
EDIT: Formatting