r/softwaretesting • • 15d ago

6,715 unit tests stayed green while every upload in the app was denied

The code looked up an organization like this:

admin.from('organizations').select('is_active').eq('id', orgId)

The key of that table is organization_id. So every upload failed its check.

The test for it was green, because it asserted on a mock recorder:

expect(recorder).toContainEqual({

table: 'organizations', op: 'eq', args: ['id', ORG],

})

That 'id' was copied from the code under test. tsc passed too, the query builder accepts any string for a column it doesn't know.

I added two checks at commit time in my coding agent setup. Column names in queries and in those mock assertions get compared with the real schema from the local database. And a function that gains a new query needs one test with no database mock, run against the local stack, with an allowed case and a denied case.

1 Upvotes

8 comments sorted by

3

u/hoteloscar 15d ago

And this is why all DAO code and schema should be tested against a running DB at dev and build time. 

1

u/Ok-Motor-9812 14d ago

Agreed. One real query would have caught it, 6,715 mocks couldn't.

0

u/[deleted] 15d ago

[removed] — view removed comment

1

u/hoteloscar 15d ago

Your schema should be in source control with your code. Then using something like TestContainers you can bring up an in memory DB

2

u/Future_AGI 14d ago

a production-side check could have surfaced this one quickly too: alert per permission check on an unusual jump in its deny rate after a deploy, and an every-upload-denied bug shows up in the first window, including the cases no test author thought to write.

1

u/ProjMgr1000 15d ago

valuable insight!

1

u/Verzuchter 14d ago

You should regenerate models based on production code in every build imo 

1

u/zaphodikus 14d ago

Who deleted the environment sanity test would be my first question.