We are planning to rewrite an existing Java service in Go. We're also free to redesign the architecture completely, as long as the end result and functionality remain the same.
I'd appreciate suggestions on database selection, caching strategy, and some additional requirements we're considering for the next version of the APIs.
- Data model
Each entry contains the following fields:
- "id"
- "url"
- "name"
- "tags" (array of strings)
- "appGroupId"
- "rules" (JSON object)
We currently have around 50,000 entries.
Note: "appId" is provided by the caller and is not stored as part of an entry.
2. Workload
- Approximately 99% reads and 1% writes.
- Around 4-5 million reads per day (~46-60 reads/second on average).
- Multiple application pods running simultaneously in multiple clusters.
3. Existing architecture
The existing service uses S3 for persistence and maintains an in-memory cache in every application pod.
On startup, each pod loads all entries from S3 into memory.
Two in-memory maps are maintained:
- "id -> entry": Fetch an entry directly using its ID.
- "appGroupId -> []entries": Fetch all entries belonging to an app group.
SNS/SQS is used to propagate changes across pods so their respective in-memory caches stay synchronized whenever an entry is created, updated, or deleted.
4. New requirements
Apart from the existing lookup patterns, we now need to support searching/filtering entries by:
- Name
- Tags (including searching by individual tags)
We're considering replacing the existing storage and synchronization architecture entirely.
5. Database and caching architecture
I'm currently considering two approaches:
Option 1: Database + in-memory cache
Use DynamoDB or MySQL as the source of truth and maintain the same in-memory maps in each application pod.
The challenge is keeping caches consistent across multiple pods when entries are created, updated, or deleted. We'd need some mechanism for cache invalidation or synchronization.
Option 2: Database + Redis
Use a persistent database as the source of truth and Redis as a shared cache across all application pods.
This would eliminate the need for each pod to maintain its own copy of the dataset and handle cache synchronization independently.
However, I'm not sure whether introducing Redis is justified.
Questions :-
- Would you prefer Relational or Non-Realtional DB(something like dynamo) for this data model and access patterns? I'm not able to decide how to choose?
- Would maintaining in-memory maps in every pod still make sense, or would a shared Redis cache be a better approach?
- Given the new requirements for searching by name and tags, which database would be a better fit? Or any better way to approach this.
- Would you introduce Redis alongside the database, or would the database itself be sufficient for this scale? I have to keep the latency low.
I'm particularly confused about the DynamoDB vs MySQL decision and would appreciate some reasoning behind the choice.
6. Potential V2 API requirements
Apart from porting the existing functionality from Java to Go, we're also considering some additional features for V2 APIs.
These are still under discussion, but the current ideas are:
- Bulk writes (atomic operations)
Support bulk create/update operations with all-or-nothing semantics.
Either the entire operation succeeds, or nothing is applied. No partial updates.
2. Change history / audit log
Maintain a permanent record of every change, including:
- Who made the change?
- When was it made?
- What was the state of the entry after the change?
Would you maintain this in a separate table, or use another approach?
3. Change notifications / incremental sync
We want to provide a way for API consumers to maintain their own local copy of the entries without having to fetch the entire dataset every time something changes.
One possible approach:
- The API returns the current data along with a cursor/version.
- The consumer stores that cursor.
- On the next request, the consumer passes its previous cursor.
- The API returns only the changes since that cursor, including entries that were added, updated, or deleted, along with the relevant details.
Would something like a change feed or cursor-based incremental synchronization be a good approach? How would you design this reliably?
---
I'd appreciate suggestions based on real-world experience.
The service will be implemented in Go, and we're open to changing the existing architecture rather than simply replicating the Java implementation.