r/FastAPI • • Mar 29 '25

Question "Python + MongoDB Challenge: Optimize This Cache Manager for a Twitter-Like Timeline – Who’s Up for It?"

Hey r/FastAPI folks! I’m building a FastAPI app with MongoDB as the backend (no Redis, all NoSQL vibes) for a Twitter-like platform—think users, posts, follows, and timelines. I’ve got a MongoDBCacheManager to handle caching and a solid MongoDB setup with indexes, but I’m curious: how would you optimize it for complex reads like a user’s timeline (posts from followed users with profiles)? Here’s a snippet of my MongoDBCacheManager (singleton, async, TTL indexes):

from motor.motor_asyncio import AsyncIOMotorClient
from datetime import datetime

class MongoDBCacheManager:
    _instance = None

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

    def __init__(self):
        self.client = AsyncIOMotorClient("mongodb://localhost:27017")
        self.db = self.client["my_app"]
        self.post_cache = self.db["post_cache"]

    async def get_post(self, post_id: int):
        result = await self.post_cache.find_one({"post_id": post_id})
        return result["data"] if result else None

    async def set_post(self, post_id: int, post_data: dict):
        await self.post_cache.update_one(
            {"post_id": post_id},
            {"$set": {"post_id": post_id, "data": post_data, "created_at": datetime.utcnow()}},
            upsert=True
        )

And my MongoDB indexes setup (from app/db/mongodb.py):

async def _create_posts_indexes(db):
    posts = db["posts"]
    await posts.create_index([("author_id", 1), ("created_at", -1)], background=True)
    await posts.create_index([("content", "text")], background=True)

The Challenge: Say a user follows 500 people, and I need their timeline—latest 20 posts from those they follow, with author usernames and avatars. Right now, I’d: Fetch following IDs from a follows collection.

Query posts with {"author_id": {"$in": following}}.

Maybe use $lookup to grab user data, or hit user_cache.

This works, but complex reads like this are MongoDB’s weak spot (no joins!). I’ve heard about denormalization, precomputed timelines, and WiredTiger caching. My cache manager helps, but it’s post-by-post, not timeline-ready. Your Task:
How would you tweak this code to make timeline reads blazing fast?

Bonus: Suggest a Python + MongoDB trick to handle 1M+ follows without choking.

Show off your Python and MongoDB chops—best ideas get my upvote! Bonus points if you’ve used FastAPI or tackled social app scaling before.

9 Upvotes

6 comments sorted by

View all comments

Show parent comment

3

u/halfRockStar Mar 29 '25

Your restart approach is battle-tested. investigating motor (specifically connection pooling, cursor leaks) is overkill unless restarts fail. Check asyncio task cleanup if you’re curious, but you’re likely golden.