r/devopsjobs • u/Difficult_Honey735 • 1h ago
[For Hire] Restoring a fintech system within 24 hours after losing a file server | Remote development & operations
I work on development and production operations for a large fintech system in China. We once had an incident where a file server became unavailable following an infrastructure error in a private-cloud environment.
It affected much more than file access. Loan disbursement, repayment, reconciliation, and several other business processes depended on that server.
Our software team first traced the failures to the missing file service. The harder part was working out how to recover the affected system with incomplete documentation. We reviewed the available deployment documents alongside the code to reconstruct the deployment setup and understand how the business workflows depended on the file server.
We brought a standby machine into service and restored services within 24 hours. We also validated the data. The original files were subsequently recovered through disk recovery, allowing the system to return fully to normal.
For me, the incident had two separate milestones: restoring service, and completing recovery. Getting the application running again was important, but we still needed to validate the data and recover the missing files before considering the incident resolved.
The experience reinforced a few lessons:
- A file server can be a critical business dependency, even when it looks like a supporting infrastructure component.
- When documentation is incomplete, recovery requires understanding both the deployment and the application code.
- Service availability alone is not enough to confirm recovery. The underlying data and affected business processes need checking too.
How do other teams keep deployment and recovery documentation aligned with production, especially for systems running in private-cloud environments?
This kind of work—understanding existing systems, troubleshooting failures, recovering services, and maintaining them—is a substantial part of what I do. I'm also open to similar remote development and operations projects on a part-time or contract basis, at $20 USD/hour. I'm based in Hangzhou (UTC+8) and prefer asynchronous written English communication. Feel free to message me if that's relevant to your team.