Checkmarks Aren't Evidence: Rebuilding the Vendor Security Review
Photo by Photo by Power Digital Marketing on Unsplash on Unsplash
Ask a room of security and procurement people how they vet software vendors and you will hear a version of the same ritual. A 200-question spreadsheet goes out. A vendor returns it with every box checked and a note about their commitment to security. Someone reads it, someone signs off, and the relationship moves to the contract stage, where the security schedule gets reduced to an appendix nobody reads again.
The problem is not that the process is lazy. It is that the process is measuring the wrong thing: how well a vendor can fill out a form. What you actually want to know is how their system behaves on a bad Tuesday in March, two years after the sales engineer stopped returning your calls.
Self-Attestation Is a Confidence Trick With Good Manners
A questionnaire asks a vendor to describe themselves. That is fine as a starting point and useless as a control. Any organization can write yes next to encryption at rest. The questions do not distinguish between a company that encrypts the one production bucket and one that encrypts everything because the pipeline makes it the default.
What makes this worse is the annual cadence. A completed review from last spring tells you about last spring's architecture. Since then the vendor has shipped three acquisitions, a new subprocessor in a different jurisdiction, and a rebuild of the authentication layer. Your file says compliant. Your file is a museum exhibit.
Ask for Artifacts, Not Adjectives
The single highest-leverage change is to stop accepting narrative answers and start requesting objects. Not summaries of objects. The objects.
- A current SOC 2 Type II report, including the exceptions section, not just the opinion letter
- The most recent penetration test summary with finding counts and remediation timelines, redacted as needed
- A live subprocessor list with data location and contract terms
- An access control matrix showing who internally can reach production data
- A sample audit log export, so you can see what a log line actually contains
- Their incident response runbook table of contents and last two post-incident reviews, sanitized
If a vendor cannot produce these in a week, that is itself the answer. Note it and move on. The vendors who have their house in order will send them without a fight, sometimes before you ask.
Put the Claims in a Sandbox
Everything above is still documentary. Claims deserve a test. Ask for a sandbox tenant with synthetic data that looks like yours and spend an afternoon breaking it.
Try to invite a user outside your domain and see what happens. Attempt a bulk export as a low-privilege account. Check whether an API token created by an admin is scoped or god-mode. Look at session timeout behavior and whether it can be configured or only described as configurable. Verify that deleting a record actually deletes it, including from the search index and the backup you were told about.
You will learn more in two hours of poking than in two weeks of email. You will also learn something about the vendor's support responsiveness, which is a security control whether or not anyone calls it that.
Make the Promises Survive the Renewal
Security commitments die in the gap between the review and the master agreement. Fix that by moving the important ones into the contract itself, with teeth attached.
A workable structure usually includes: notification windows for material changes in architecture or subprocessors, a right to re-test or receive updated artifacts annually, defined remediation deadlines tied to finding severity, and a remedy clause that escalates if those deadlines slip twice. Add a clause that requires notice before a change that would alter the data residency you depended on. None of this is exotic. It is just rarely asked for, because the security team finishes before the legal team starts and the two never compare notes.
One practical note for US buyers: state-level privacy obligations and sector rules like HIPAA or GLBA often impose specific contract terms. Route your security asks through whoever owns those obligations, or you will end up with a strong security schedule that quietly conflicts with a data processing addendum.
Monitor After Go-Live, or Start Over
The review is not a gate. It is a subscription. Build a light but relentless post-signature habit:
- Re-pull SOC 2 and pen test artifacts on a calendar, not on request
- Watch the subprocessor list for changes and read the diff
- Track their status page and incident history; a vendor with no incidents is often a vendor with no disclosure
- Review their release notes for anything touching auth, permissions, or data export
- Re-run a slice of your sandbox tests once a year, especially after a major version
Whoever owns each vendor in your stack should have a name and a date next to it. If the answer to who is responsible for a given vendor is a team name, you do not have ownership. You have a shared assumption.
The Point of All This
Security theater is comfortable because it produces a document that looks like diligence. Evidence is uncomfortable because it occasionally tells you the vendor you already like is not ready. Better to find that out in the sandbox than in the incident channel. The questionnaire is not going away, and it does not need to. It just needs to stop being the last thing anyone checks.