After a year with models, I am returning to a gap they do not close
It is 15 November 2023. A year ago, in my 2022 summary, I recorded ChatGPT as the headline at the end of the year, not as the tool of the year. Today I have to admit that the headline stayed with me: almost everything I wrote on this blog in 2023 concerned language models in QA work. Test cases, prompt structure, synthetic fixture data, reading flaky runs, documentation drafts. I do not regret a single one of those evenings.
Yet throughout that year, my test suite had a gap that none of those experiments even touched: the security of the application I am testing. I do not mean who has access to the repository, or whether an API key is kept in an environment variable, because I have those things in order. I mean something simpler and more uncomfortable. My E2E suite stays green when a user does what I anticipated, but none of us has ever checked how the same application responds when an authenticated user asks for a resource that belongs to someone else. A green E2E run does not mean “there is no IDOR.” It only means I have no test looking for one.
Before anyone decides this is a post about a fashionable topic: it is not. I have seen prompt injection and language model jailbreaks on conference slides since spring, but that is not this post. Nor is this a pentesting course, because I do not know how to conduct a pentest and do not sell one. This is a note about what a test automation engineer can do with security basics before anyone calls it a service: use the OWASP Top 10 as a risk map, run a ZAP baseline as CI smoke, and use Burp manually around login.
Two technical notes make the context clear. Yesterday, 14 November, .NET 8 was released. I am noting it and returning to the subject because nothing below depends on the runtime version. My projects use .NET 7, and the examples remain on 7. I am pinning Playwright to 1.39.0, the same version I used a month ago.
A risk map, not a checklist of four hundred tools
My first instinct was the same as it always is with a new subject: look for a tool. After two evenings with lists of “top security testing tools,” I know that this is the worst possible place to start. There are hundreds of tools, each one wants to be a platform, and none answers the question I actually have: which risks apply to the application I am testing this week?
So I begin with a map, and that map is the 2021 edition of the OWASP Top 10. The previous edition came out in 2017, so this is still the current list. That is the first thing to know about it: it changes once every few years, so refreshing it every sprint makes no sense. The second point matters more. It is an awareness document built from data and a survey, not a test plan, a standard, or a certificate. You cannot “pass the Top 10.” You can use it to name a risk you had not considered before.
Broken Access Control, or A01, sits at the top of the 2021 list. In the 2017 edition, this category ranked fifth. That jump is the most useful information in the entire document for me, because Broken Access Control is exactly the class of defect that neither my unit suite nor my UI-driven E2E suite sees. The UI never asks for someone else’s ID because it has no button for doing so.
I am not testing all ten categories. I choose three that fit what I already have at hand: an API and a session.
A01, Broken Access Control. Failing to verify who owns a resource, failing to check permission at the function level, such as an admin endpoint responding to an ordinary user, or trusting a field from the request instead of the identity in the token. This is where the IDOR from the previous paragraph belongs.
A07, Identification and Authentication Failures. I care about three questions for which I can write tests: does logout really invalidate the session on the server, does the token expire and what happens after it does, and does the session identifier appear in the URL? This is not the entire category. It is three testable statements taken from it.
A05, Security Misconfiguration. A default configuration carried over from the project template, a diagnostic endpoint exposed on staging, an error page containing a stack trace, wildcard CORS combined with cookies, or missing security headers. Of these three categories, this is the only one where automation truly helps me.
I deliberately leave the rest alone, and I will say why. In my stack, Injection (A03) is addressed mainly by an ORM with parameterized queries and by code review. I also have no intention of publishing payloads in a blog note. Vulnerable and Outdated Components (A06) is not a subject for an evening with an interactive tool. It needs a separate pipeline job: dotnet list package --vulnerable on the .NET side and dependency scanning on the npm side. In fact, this is the cheapest item on the whole list and the first one I would enable if I could do only one.
I want to record one sentence now so that no one can talk me out of it next year: neither the Top 10 nor any scanner replaces code review and threat modeling. A scanner cannot tell me that an endpoint should not exist at all, or that two roles were designed in a way that lets one see too much. That requires a conversation with people who understand the domain, and there is no way to automate it.
Automation: ZAP baseline as smoke, not a scan
OWASP ZAP can do a great deal, but I am interested in one of its least exciting features: the baseline. This is a passive run. The tool navigates the application, inspects what it receives in responses, and reports alerts from passive rules. It does not send requests intended to change application state, and it finishes in minutes rather than hours. That is an entirely different operation from a full active scan, which sends hundreds of requests, writes data, can empty a test user’s cart, and may wake up the alerting system.
That distinction determines where the job belongs. I return to the three buckets from my CI/CD test strategy and place the baseline in the second one, after merge. A baseline requires a deployed application, while a pull request gate should be deterministic and complete within minutes. An active scan, if I ever run one, belongs at night on an environment no one else is using at the time. Never in production, and never against someone else’s host.
Here is a sketch of the job, deliberately without report publishing or build steps:
security-baseline:
if: github.event_name == 'push'
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- name: ZAP baseline on staging
uses: zaproxy/action-baseline@v0.10.0
with:
# our own environment, our own internal domain
target: https://staging.qa.internal.example/
rules_file_name: .zap/rules.tsv
# -I: warnings do not fail the job, only FAIL makes it red
cmd_options: '-I'Three decisions in this sketch are its whole point. The trigger is push, so the job does not block anyone’s merge. There is no active mode because this is supposed to be smoke, not a scan. And there is a rules file, which in practice is the most important file in the entire exercise.
The .zap/rules.tsv file lists rule identifiers with a level: IGNORE, WARN, or FAIL. Everything is a warning by default, and with -I, warnings do not break the build. I therefore get a red job only for the rules I have raised to FAIL myself, meaning those that I judged to represent a genuinely high risk in my application after reviewing the first report. Everything else goes into the report for review instead of turning the job red.
There is one condition without which this whole setup stops working after two weeks: every IGNORE entry needs a comment with a reason and a date. A silenced alert without an explanation is not a decision. It is something forgotten. A job that remains red indefinitely will be disabled by the first person whose release it blocks. I have seen this happen with flaky tests, and the mechanism is identical.
What the baseline actually found on my own staging environment was a missing Strict-Transport-Security header, a missing Content-Security-Policy header, a cookie without the SameSite attribute, a server header exposing its version, and an outdated JavaScript library loaded from a CDN. I am not pasting the report because that is not the point. Besides, half of these alerts turned out to reflect differences between staging and production: some of the headers are added by an ingress layer that simply does not exist on staging. This was the first time I saw that a difference in environment configuration is itself a finding, not noise.
What the baseline did not find matters more. It did not find the missing resource ownership check because a passive rule has no idea who owns order number 41. To the scanner, a 200 OK response with valid JSON is exemplary. It looks exactly like a correct response should, and that is precisely what makes this class of defect different from the others.
Manual work: Burp around authentication
That is why the second half of the evening looks different: a browser, a proxy, and one request repeated once. I use Burp Suite Community Edition, which gives me the proxy and Repeater. The scanner is part of the Pro edition, and I do not need it at this stage. Setup means installing the CA certificate in the browser and applying a scope filter so that the history contains only my host, not the entire internet.
Two rules apply before the first click, and both are firm. I work only in an environment we own and only with accounts I created myself using synthetic data. I do not point the proxy at a live host I do not own, even when “it is only one request.”
The idea is simple enough to fit in one sentence, and it is not an intrusion guide because there is nothing to break into. I create two synthetic accounts, A and B. Account A has an order. I sign in as A, capture the request that reads this order, move it to Repeater, and send it once more, this time with account B’s token. The expected response is a refusal: 403, or 404 if we do not want to confirm that the resource exists. The same request, a different owner’s identity, and an expected refusal. That is all. If I receive 200 with account A’s data, the endpoint does not verify ownership, and I have a finding I cannot unsee.
The demo I describe is synthetic and exists only in this post. It uses an order query handler that checks only whether the token is valid and forgets about ownership. On the .NET side, the difference between the faulty and correct variants is one predicate:
// faulty handler: authorization ends after confirming the token is valid
var order = await db.Orders
.FirstOrDefaultAsync(o => o.Id == id);
// correct handler: the caller's identity is part of the query
var order = await db.Orders
.FirstOrDefaultAsync(o => o.Id == id && o.OwnerId == callerId);That is the whole substance of this class of defect, which is why there is no payload to publish here. There is no clever string and no bypass of the login mechanism. There is a query missing one condition, and that query behaves perfectly for as long as the person asking is the owner.
The evening with Repeater does not end with a report, however, because no one on my team cares about a report, and rightly so. It ends with a test in the API suite, the same place where I keep the integration tests described in my post about dependency injection for tests:
[Fact]
public async Task Reading_another_owners_order_returns_403()
{
var owner = await _factory.CreateSyntheticUserAsync("owner@example.com");
var stranger = await _factory.CreateSyntheticUserAsync("stranger@example.com");
var orderId = await _factory.CreateOrderForAsync(owner);
var client = _factory.CreateClient();
client.DefaultRequestHeaders.Authorization = stranger.Bearer;
var response = await client.GetAsync($"/api/orders/{orderId}");
Assert.Equal(HttpStatusCode.Forbidden, response.StatusCode);
}There is one less obvious condition for this test to be meaningful. The test host must not replace authentication with a scheme that lets everything through. The trick of swapping an implementation through the dependency injection container, which I described a year ago as a convenience, is also the easiest way to remove silently the exact mechanism I am asking about here. If my test configuration registers a permissive authentication handler, my 403 assertion is measuring my own fake. I checked this in my setup before writing a second test of this kind, and it was worse than I expected.
Playwright 1.39.0 will not catch this, and there is no reason to make it try. A UI test clicks what the UI exposes, and the UI never enters someone else’s identifier. This question belongs at the API layer, and that is where it stays.
What I already have on this blog, and what it does not test
The most uncomfortable part of this subject is that several things I have described here as good practices create a false sense of security coverage.
An HTTP mock is not a security mechanism. A WireMock stub returns exactly what I told it to return, and it never refuses. If the real provider rejects a token without the required permission scope while my stub answers every request with 200, my green suite is testing a polite fiction. A mock answers questions about availability and response shape, not access policy.
A contract does not verify permissions. A contract test from my introduction to contract testing describes the message shape and status for the interaction in question. Provider verification passes because I prepared the provider state myself, along with the caller’s identity. Who is entitled to call the endpoint is outside the contract, and no pact will catch it.
An in-memory host cannot see infrastructure. Integration tests using an application factory, which I discussed in dependency injection for tests, have no ingress layer, TLS termination, or web application firewall in front of them. All the headers added by production infrastructure are invisible in such a test, so an assertion at this level that “we have CSP” asserts nothing.
A container provides isolation, not hardening. The container suite from Testcontainers on Azure Pipelines gives me a repeatable environment and a clean state. It does not provide a threat model or change the fact that an image with a default password in an environment variable is exactly what it appears to be. This is a tool for test isolation. Its presence in a pipeline is not an argument in a security discussion.
None of those posts was wrong. Each answered a different question, and only now do I see where it stops answering.
Summary: what QA can catch and what requires a separate agreement
After two weeks of reading and two evenings with the tools, I have a list of what I consider security basics for myself today, and it is shorter than I feared.
A ZAP baseline as a post-merge job, red only for rules I raised myself, with a suppression file where every entry has a reason and a date. A handful of resource authorization tests in the API suite: the same request, a different owner, and an expected refusal. Three testable session questions: does logout invalidate it on the server, does the token expire, and is the session identifier absent from the URL? Dependency scanning as a separate, inexpensive job. And one process finding: staging configuration that differs from production is a finding, not noise.
There are also things I do not do and will not call by another name. I do not call this a pentest because a pentest involves a methodology, a scope, and people who do it professionally. I do not touch systems we do not own. I do not build exploit chains or publish findings, even from our own systems. A pentest and a bug bounty program require a separate agreement: scope, rules of engagement, and written permission before the first request. Threat modeling remains with the people who know the domain, and code review remains with the team.
The sentence I want to remember from this November is that a green E2E run tells me the application works for a well-behaved user. Only a test that asks the question can tell me whether it also works for a user who enters someone else’s ID. A language model will not write that test in place of thinking about risk, because someone first has to decide which resource and which owner to ask about.
Next month I will sit down to write the year-end summary, and I already know it will be about AI because that is what this year has been about. This note exists so that the summary does not omit one paragraph about the part of my test suite that models did not touch at all.