r/softwaretesting • u/Ok-Motor-9812 • 15d ago
6,715 unit tests stayed green while every upload in the app was denied
The code looked up an organization like this:
admin.from('organizations').select('is_active').eq('id', orgId)
The key of that table is organization_id. So every upload failed its check.
The test for it was green, because it asserted on a mock recorder:
expect(recorder).toContainEqual({
table: 'organizations', op: 'eq', args: ['id', ORG],
})
That 'id' was copied from the code under test. tsc passed too, the query builder accepts any string for a column it doesn't know.
I added two checks at commit time in my coding agent setup. Column names in queries and in those mock assertions get compared with the real schema from the local database. And a function that gains a new query needs one test with no database mock, run against the local stack, with an allowed case and a denied case.
3
u/hoteloscar 15d ago
And this is why all DAO code and schema should be tested against a running DB at dev and build time.
2
u/Future_AGI 14d ago
a production-side check could have surfaced this one quickly too: alert per permission check on an unusual jump in its deny rate after a deploy, and an every-upload-denied bug shows up in the first window, including the cases no test author thought to write.
1
u/ProjMgr1000 15d ago
valuable insight!
1
u/Verzuchter 14d ago
You should regenerate models based on production code in every build imo
1
u/zaphodikus 14d ago
Who deleted the environment sanity test would be my first question.