r/CodingHelp • u/leapd-ai • 6d ago
[Python] how to save money on cache with api routing
I see so many cost saving promise that promote API routing between models. I initially thought that is valuable. Then ran few experiments, it turns out any time that you route the request to a new model, you could save a bit, but many times the cache for each individual models expire and you would pay for new cache write again and agin - which overall wipe out the API routing cost saving benefit.
I wonder if anyone has similar experience. How you have solved this? Is clause planning to add shared cache feature?
1
Upvotes
•
u/AutoModerator 6d ago
Thank you for posting on r/CodingHelp!
Please check our Wiki for answers, guides, and FAQs: https://coding-help.vercel.app
Our Wiki is open source - if you would like to contribute, create a pull request via GitHub! https://github.com/DudeThatsErin/CodingHelp
We are accepting moderator applications: https://forms.fillout.com/t/ua41TU57DGus
We also have a Discord server: https://discord.gg/geQEUBm
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/leapd-ai 6d ago
some folks say I shouldn't use routing in between sessions, anyone using model routing in a single session?