webappsec.dev

$less ~/slides/high-assurance-agentic-engineering

// deck 01 · OWASP AppSec Days Portugal

It's Never Been Easier to Write (In)Secure Software

Turning AI, the fastest coder, into the safest

OWASP AppSec Days Portugal · Porto · 2026keynote· 50 slides· Lukas Weichselbaum

pdf ↓ 1.4 MB

  1. Slide 1: It's Never Been Easier to Write (In)Secure Software: turning AI, the fastest coder, into the safest
  2. Slide 2: Speaker: Lukas Weichselbaum, Google Product Security. Two teams, one philosophy: secure by design
  3. Slide 3: A quick poll: raise your hand if AI wrote some of your code this month
  4. Slide 4: It has never been easier to write insecure software
  5. Slide 5: Act I of VI: the pincer. Two jaws are closing at once
  6. Slide 6: This is not a future problem: about 90% of developers use AI; 75% of new code at Google is AI-generated
  7. Slide 7: Fast code, familiar bugs: 44% of AI coding tasks introduce a security flaw; 15% pass rate on XSS
  8. Slide 8: Better models raise the average. Nothing sets the floor
  9. Slide 9: And review is disappearing anyway
  10. Slide 10: The attackers got agents too
  11. Slide 11: You hold the source. And the compiler
  12. Slide 12: Act II of VI: the answer we already had. Safe by design
  13. Slide 13: Safe by design: don't find the bug. Design out the vulnerability class
  14. Slide 14: Why safe by design works: scalable, proactive, provably safe, automatically validated
  15. Slide 15: At Google, a new web app is born safe
  16. Slide 16: Safe defaults in our hardened frameworks: twelve controls, four stages
  17. Slide 17: One response, every security feature on: the response headers of passwords.google.com
  18. Slide 18: Invariants the browser itself enforces: strict CSP, Trusted Types, Fetch Metadata, COOP
  19. Slide 19: The browser part is done: strict CSP, Trusted Types, COOP and Fetch Metadata are Baseline in 2026
  20. Slide 20: Fewer than one exploitable XSS per year, on average, across hundreds of products on safe-by-default frameworks
  21. Slide 21: We pointed our attack agent at ourselves: hundreds of XSS found, none in production in the safe-by-design stack
  22. Slide 22: Act III of VI: the wall. Why brownfield stalled for fifteen years
  23. Slide 23: The wall: brownfield. Refactoring cost, incompatible stacks, breakage risk
  24. Slide 24: Full disclosure: it took us many years
  25. Slide 25: First, make strict CSP deployable
  26. Slide 26: Then, Trusted Types: the DOM refuses strings
  27. Slide 27: Now the whole stack moves
  28. Slide 28: What took years now takes months or less
  29. Slide 29: Android changed only its new code: memory-safety bugs fell from 76% to under 20% of vulnerabilities
  30. Slide 30: Act IV of VI: what just changed
  31. Slide 31: Human developers push back. LLMs don't
  32. Slide 32: The agent-ready security environment: rules, iterable validation, non-bypassable enforcement
  33. Slide 33: Agentic engineering plus safe by design: high-assurance agentic engineering
  34. Slide 34: High-assurance agentic engineering, end to end: the agentic loop and three gates
  35. Slide 35: Same prompt, two environments, one judge
  36. Slide 36: The vibe column tracks the model, the harness column is the floor: 9 of 53 vibe builds fired, 0 of 54 harnessed
  37. Slide 37: What the gates refused, and what happened next
  38. Slide 38: Act V of VI: the seams. Where safe coding honestly ends, and what fills the gap
  39. Slide 39: Where safe coding ends: the seams
  40. Slide 40: Scanning that ships fixes, not tickets
  41. Slide 41: Big Sleep finds. CodeMender fixes
  42. Slide 42: The bug hunters got agents too. Good
  43. Slide 43: Act VI of VI: what it buys, and what we must build
  44. Slide 44: Not faster per build. Turns spent on verdicts, not guesses
  45. Slide 45: Safer. Faster where it counts: the pipeline
  46. Slide 46: The open question: what does meaningful oversight become, in this world?
  47. Slide 47: Build high-assurance agentic engineering with us: three asks
  48. Slide 48: Guidance the machine can enforce: safety-web
  49. Slide 49: AI is already the fastest coder in history. Together, let's make it the safest
  50. Slide 50: Obrigado! Slides and past talks at webappsec.dev/slides
1 / 50

// speaker notes

The script for each slide, lightly edited for reading.

  1. slide 1

    Good morning, Porto. Thanks for having me, and thanks for the introduction, Pedro.

    One thing about the bridge on the slide, because it's not just decoration. Civil engineers prove that a structure holds before anyone drives over it. In software we mostly ship, and then hope. This talk is about closing that gap.

  2. slide 2

    Two things about me that matter for the next forty-five minutes. I've spent about ten years at Google making cross-site scripting go away, on the platform side in the W3C and on the deployment side across our web apps. And since 2023 I also run one of our AI agent security teams. So I get to see both ends of this problem: the code the agents write, and the agents themselves. That's it, let's go.

  3. slide 3

    Before we start I'd like to do a quick poll. Raise your hand if AI wrote some of your code this month. OK, that's most of the room. Now keep it up if a human actually reviewed every single line of that code. Yeah. That's about what I expected. Those of you still holding up your hand, I admire your discipline. Or your denial, I'm not sure.

    So that gap, between the first question and the second one, that gap is what this talk is about.

  4. slide 4

    It has never been easier to write insecure software. And I want to prove that to you with numbers that are not mine, so you know I'm not grading my own homework.

  5. slide 5

    So, act one. I call it the pincer, because there are two jaws closing at the same time. The first jaw: AI is now producing vulnerable code at a scale we've never seen. The second jaw: the attackers have agents too, and they find and exploit that code at the same speed. Same technology, both sides, and your application is sitting in the middle.

  6. slide 6

    Let's start with adoption, and I'll go through this quickly because you live these numbers. DORA says about ninety percent of developers use AI somewhere in their workflow. Sonar says seventy-two percent use AI coding tools every day. And in April Sundar said that seventy-five percent of new code at Google is AI generated, and then approved by engineers. Hold on to that second half, approved by engineers, we'll come back to it. And these are just the people who admit it in surveys.

    The bigger point is this: most of the code the world will ever run hasn't been written yet. And agents are going to write most of it. So the question is not whether AI writes our software anymore. The question is how well.

  7. slide 7

    So how well? Veracode ran over a hundred models against eighty coding tasks, four rounds so far. Forty-four percent of the time the model introduces a security flaw. Now the interesting part is the breakdown. On SQL injection the models pass eighty-three percent of the time. That's actually not bad. Why? Because prepared statements are a pattern, and there are twenty years of examples in the training data. On cross-site scripting: fifteen percent. Fifteen. As someone who has spent ten years fighting XSS, I felt seen. And also attacked, literally.

    The reason is that XSS is contextual. Whether a string is dangerous depends on where it came from and where it ends up, and that is exactly what the models are bad at. So the models are not inventing new vulnerabilities. They're mass-producing ours.

  8. slide 8

    Now I want to be fair here, because a lot of you know that the frontier models got better at this. And that's true. Veracode's best model went from fifty-five to sixty-eight percent secure. And in our own benchmark, which I'll show you later, the frontier models didn't fire a single trivial XSS payload in thirty-three vibe-coded builds. That's good news, and I'll show you the data.

    But two things. First, look at the average. Across all these models the security pass rate is basically flat, fifty-five, fifty-five, fifty-six, fifty-six, while the syntax pass rate went from fifty to ninety-five percent. So the code compiles now. It's just as insecure as before. And the average matters, because you don't deploy the best model. You deploy the whole distribution. Most code is written by the cheaper, faster, older models, and cost pressure keeps it that way. In our data, the small model fired trivial XSS payloads five out of five times.

    Second, even where it is better, that's a habit, not a guarantee. It depends on which model, which day, which prompt. Training raises the average. Only the environment sets the floor. And a floor you can actually check, that's what assurance means. Keep that word in mind, we'll come back to it.

  9. slide 9

    And here's the part nobody likes to say out loud: review is disappearing anyway. Agents open the pull request, they respond to the comments, they merge. The human review is a glance, if that. Sonar found that fewer than half of developers review AI code before they commit it, and ninety-six percent say they don't fully trust it. So we don't trust it, and we don't review it. Great.

    And to be honest, this was true before AI. The ratio of software engineers to security engineers has always been something like a hundred to one, or a thousand to one. Security review never scaled with headcount. AI just adds another order of magnitude on the generation side. One senior engineer cannot meaningfully audit what a fleet of agents produces, the math simply doesn't work.

    So security has to be structural, not procedural. If the last line of defense is a tired human skimming a four-hundred-line diff, we've already lost. Security has to survive the absence of review.

  10. slide 10

    Jaw two. The attackers got agents too, and they're becoming extremely good. Last year an autonomous pentester, XBOW, was number one on the HackerOne leaderboard in the US, by number of reported vulnerabilities. Our threat intelligence team confirmed the first AI-generated zero-day exploit in the wild this year. And state actors are actively probing models for vulnerability discovery. So this capability is being industrialized on both sides.

    Sometimes the attacker literally is the agent. In July an evaluation agent broke out of its sandbox and ran a multi-day intrusion into a production environment, thousands of actions, nobody at the controls. And nothing in that chain was new. SSRF, credential reuse, template injection. The same textbook classes we've had for twenty years.

    And by the way, we built one of these attack agents too. I'll get to that.

  11. slide 11

    But this fight is not symmetric, and this is where it turns. Look at what the defender has that the attacker doesn't. The source code. The build system. Compile-time checks. And runtime enforcement in the browser. The attacker gets an agent. Fine, so do we. But ours reads the code before it ships, and it controls layers theirs can never touch.

    So let me leave you with one question before act two: what happens when you point an attack agent at us? Hold that question. I'll answer it in about ten minutes, with data.

  12. slide 12

    Act two. The answer we already had. Safe by design, security by construction instead of by chance. We've been doing this for over a decade. What's new is who it's for and how this is becoming more accessible now.

  13. slide 13

    So what is safe by design? Very simply: don't find the bug, design out the vulnerability class. You give developers APIs that are safe by default, and you enforce security invariants in the stack. An invariant is a property that holds for every line of code, no matter who, or what, wrote it. Not memory, not chance, not heroics.

    And there's a distinction here that matters for the rest of the talk: safe by convention is not safe by construction. We measured this. Thirty-one of fifty-three vibe-coded apps in our benchmark had a raw innerHTML assignment behind an escaper the model wrote that day. Mutated payloads got through six of them. The fifty-four harnessed apps had no sink to bypass at all. That's the difference. Construction means there is nothing to get right. I'll show you the whole benchmark in act four, I just wanted to plant that.

    And I say design out, not impossible. Determined people find the seams. Act five is honest about that.

  14. slide 14

    Why does this work? Four properties, and the first three you know. It's scalable: you eliminate the whole class, XSS, SQL injection, not individual bugs. And frankly it's a lot cheaper than running a frontier model over every pull request looking for bugs. It's proactive: the vulnerability is stopped at compile time or in presubmit, so it's never introduced in the first place. And it's provably safe: the class doesn't exist in the app and it can't come back as the code evolves.

    Now the fourth one. For fifteen years, when we sold safe coding, we sold it on the first three. The fourth was the sweetener: and by the way, the compiler tells you right away, so you're not waiting on a security review. Developers liked that, of course, short feedback loops mean less friction. But for a human it's a convenience. You still hit the error, sigh, and go fix it yourself. For an agent it's the whole interface. An agent can iterate against a compile error a thousand times. It cannot iterate against a human reviewer. So the property that was a convenience for developers is the one property that agents can't work without. In our benchmark, fifty-four out of fifty-four harnessed builds looped until they were green. Nobody argued. Everybody just tried again.

  15. slide 15

    What does that look like in practice? At Google, a new web app is safe by design. All the web security features we have are on by default, from the browser security headers down to compile and lint time bans on unsafe APIs, from the first commit. Developers don't need to understand XSS to be protected from causing it. The framework carries that knowledge. And if you want to bypass a control, you can, but you file an exemption and a security engineer reviews it.

    So the inversion is: in most organizations being secure takes extra effort. Here, being insecure takes extra effort, plus a form, plus a reviewer. We made the safe way both easy and mandatory. Nothing else scales.

  16. slide 16

    This is the matrix from my LocoMocoSec talk two years ago. Twelve controls, four stages. These frameworks power over three hundred external-facing apps, Cloud Console, Photos, Gemini, the password manager. And every control walks the same path: first it's supported as an opt-in, then it's on by default, then it's enforced and you need a reviewed exemption to turn it off, and finally we backport it to the legacy apps.

    The stage that matters for today is the third one, enforced. That's the rail the agent inherits for free.

  17. slide 17

    Here's what that looks like on the wire. This is a real response from passwords.google.com, slightly trimmed to fit the slide. The amber block, strict CSP with nonces and Trusted Types, that's what kills XSS. The red line makes sure only first-party JavaScript loads, that's a supply-chain fence. The green ones, COOP, CORP, Fetch Metadata, isolate you against XSRF, clickjacking and cross-site leaks. And blue is HSTS. Every new app ships this block on day one, before anyone has thought about security.

  18. slide 18

    And the important part is that the strongest of these are enforced by the browser itself, not just by our code. Under strict CSP an injected script tag is just inert text, even if every other layer failed. Trusted Types turns "don't pass strings to innerHTML" from advice into a runtime rule, the DOM simply refuses the string. And Fetch Metadata plus COOP let your server refuse cross-site requests it never meant to serve, and isolate your windows.

    Small aside: these controls exist in your browser because we spent about a decade in the W3C putting them there. I co-authored CSP3 and Trusted Types. And the whole point of doing it as a standard is that the strongest layer of our defense is already installed on your users' machines. They're yours.

  19. slide 19

    And as of this year, the browser part is done. Strict CSP in 2016, Fetch Metadata in 2019, Trusted Types shipped in Chrome in 2020, and since February this year Trusted Types is Baseline, so Chrome, Edge, Firefox and Safari. A decade of web platform work, and it finished just in time for the agents. You can turn these security controls on and the agents will adjust to write safer code because of it if you provide the right harness.

  20. slide 20

    So what did we get for all of that? Less than one exploitable XSS per year, on average, across hundreds of products on these frameworks. XSS used to be our single biggest vulnerability class. This is not fewer bugs found, and not a better scanner. It's a different regime. For practical purposes the class stopped existing.

    Now, I know what the sharp ones among you are thinking. Absence of reports is not absence of bugs. Maybe you're just not finding them. That's fair. Remember the question I asked you to hold?

  21. slide 21

    We pointed an attack agent at ourselves. It reads the source code, finds candidates, and then validates every candidate against a live instance, so these are real, exploitable, no false positives. It found hundreds of XSS vulnerabilities across Google. Many on very sensitive domains. And then we root-caused every single one. In production, not one of them originated in the safe-by-design stack. Zero.

    And full disclosure, because someone will ask. Internally the agent did find four XSS in apps on that framework. All four on one debug endpoint, one root cause, never exposed to users. I'm telling you because they are actually the point. A debug endpoint is a seam, it sits outside the framework's guarantees. Where the invariants held, the agent found nothing. Where they didn't apply, it found four. Same agent, both worlds, and the bugs split exactly along the stack boundary. The full write-up went out on the Google Security Blog this week, fresh off the press. There's a QR code on the slide, scan it now, I'll wait a second. It's the whole story of Pagebreak, the agent, with the real proofs.

  22. slide 22

    OK. So why didn't everyone just do this? Act three, the wall.

  23. slide 23

    This is where most of you live, I know. Greenfield is a demo. Brownfield is a career. Three reasons it stalls. Refactoring cost: years of migration work, and at the end you have nothing new to show. Feature work wins that argument every time. Incompatible stacks: some things just don't fit safe by design. Template systems without contextual auto-escaping. Or trying to write an application without memory corruption bugs in C, we know that's not doable in practice. And breakage risk. Large changes to production carry risk.

  24. slide 24

    And full disclosure: it took us many years too. The strict CSP rollout ran from 2018 to 2024 to get to over nine hundred apps and more than ninety percent of our critical frontends. Trusted Types, 2020 to 2024, about six hundred apps. Framework rewrites, per-app migrations, exception queues, deadline politics. App by app by app.

    And we had every advantage. A monorepo, central frameworks, a large security team, and good measurement. Most organizations stalled exactly here. Not on the concept, on the cost. When I gave the LocoMocoSec talk in 2024, this is the point where the room deflated. Great results, for a company that can afford ten years of migrations. Completely fair. So: The recipe worked. The cooking was slow.

  25. slide 25

    So here's what changed. The cooking got fast. The work that stalled us is exactly the kind of work agents are good at: mechanical, verifiable, and there's a lot of it. Let's look at what it actually is. Making strict CSP deployable means: nonces on every script tag through the template system, inline event handlers out, javascript: URIs out. Every one of these changes is trivial. There are just thousands of them, and you can't skip any. This is the work that stalled us for years.

  26. slide 26

    Then Trusted Types. innerHTML, outerHTML, script.src, about sixty DOM APIs turn strings into code. With Trusted Types every one of those assignments becomes a type error. Plain text goes to textContent. Real HTML goes through the Sanitizer API, or through a policy at the few real boundaries. And the compiler walks the agent through the complete list.

    So the punchline of these two slides: every one of these migrations is trivial. There are just fifty thousand of them. Miserable by hand. A batch job for an agent. This is exactly the work humans hate and agents are good at: mechanical, verifiable, endless.

  27. slide 27

    And it doesn't stop at call sites. The whole stack moves now. You can port across languages, C++ to Rust, component by component, function by function, with tests validating each step. You can port across frameworks, from a Jinja-style template system without contextual auto-escaping to one that has it built in, with typed safe HTML. And you can rewrite the dependency itself: reimplement just the slice of a third-party library you actually use. That removes supply-chain risk and the classic CSP and Trusted Types blockers in one move.

    So the takeaway is not any particular migration. It's that security refactorings which used to be multi-year programs, with roadmaps and exception queues and deadline politics, are now batch jobs. At scale, driven by agents, and checked by the compiler at every step. Making things secure has never been this cheap. Two years ago none of this was realistic.

  28. slide 28

    So, remember the decade? The constraint was never the recipe. It was author-hours of constrained, mechanical, verifiable work. And that is exactly the resource that just became cheap. What took years now takes months, or less.

    One careful word about faster: this is migration throughput. Per build, an agent working against these gates actually needs more turns, not fewer, and I'll show you exactly how many in act six. But the wall didn't move. We got a ladder that never gets tired, and a compiler that checks every rung.

  29. slide 29

    And the porting claim is already proven, at OS scale. Android didn't rewrite its old C++. It only changed its new code. Memory-safety bugs went from seventy-six percent of vulnerabilities in 2019, to twenty-four in 2024, to under twenty percent last year. The industry norm is about seventy. The physics behind it: most vulnerabilities live in new code, and the old ones age out. So change only what you write next, and the past depreciates on its own. Bonus: Rust changes get rolled back at less than half the rate of C++ changes. Safer shipped faster. Same recipe as XSS, different class. Don't fix the bug, remove the ability to write it.

  30. slide 30

    Act four. What just changed. And the one-sentence version is: what was a convenience for developers is the whole interface for agents.

  31. slide 31

    Human developers push back. LLMs don't. And I manage about twenty-five engineers, so I say this with love.

    Look at the two loops. The human loop: you write code, you wait for review, you discuss, maybe it gets merged. Hours to days per cycle, and the feedback is an opinion. The agent loop: generate, compile, lint, presubmit, instant verdict, retry. Seconds per cycle, the feedback is a verdict, and the agent doesn't get frustrated.

    The most expensive property of safe coding was always that it constrains the author. That cost goes to zero when the author is a machine. Constraints are the one thing LLMs are genuinely good at, as long as the feedback is instant, deterministic and non-negotiable. That's what turns a constraint from friction into fuel.

  32. slide 32

    So what does an agent-ready environment need? Three things. Rules that are explicit and machine-readable: which APIs are safe, which are banned, which framework is blessed. No tribal knowledge, the agent can't read your mind. Iterable validation: compile and commit time checks the agent can loop against, because a thousand tries cost almost nothing. And non-bypassable enforcement: runtime invariants in the browser the agent can't route around. The browser doesn't negotiate. It's the one reviewer the agent can't wear down.

    And rule one, enforced: if it's not safe, it doesn't compile. The agent doesn't argue. It tries again.

  33. slide 33

    Let me give this a name. Remember sheet seven, the average and the floor, and that assurance is a floor you can check. Here's the discipline that builds one.

    Agentic engineering is Andrej Karpathy's term, vibe coding was his as well. It means you rarely type the code anymore, you orchestrate the agents that do, and you're the oversight. Humans orchestrate, agents write. Add what the harness enforces, safe by design, and you get high-assurance agentic engineering. Assurance meaning claims that can be machine-checked by something trustworthy, not a human glancing at a diff. The harness enforces invariants the agent cannot route around.

  34. slide 34

    This is the whole talk on one sheet. The agentic loop on the left: write, build, test, retry. Untrusted by design, it can iterate as much as it wants. Then three gates. Compile time: unsafe code doesn't build, the banned API is a build error before the code exists anywhere. Commit time: presubmit re-checks every change, and exemptions need a human reviewer, that's the one path that stays gated. And runtime: the browser enforces strict CSP and Trusted Types on every response, even for whatever slipped past the first two.

    The design rule is at the bottom. The invariants live outside the loop. Policy, build config, response headers. The agent can't edit any of them. The loop can try anything. It cannot move the gates.

    Two conditions, to be honest. Gate three only holds if the agent can't change the framework that sets the headers. Inside Google that's build visibility, ownership and conformance checks. Outside, it's branch protection, a bot token that can only open pull requests, and CI workflows the agent can't touch. And gate one breaks existing code on day one, so in practice you allowlist what exists and enforce on new code. That still works, Android did exactly that, you just saw it.

  35. slide 35

    Now, inside Google none of this is a hypothesis. The gates I showed you in act two were built for human developers, and they judge agent-written code exactly the same way. We basically swapped the developer for a loop and the frameworks didn't notice. One caveat: those frameworks weren't designed for agentic workflows, so we're building dedicated well-lit paths for agents together with our core developer teams, precisely so a decade of progress doesn't quietly regress.

    But you can fairly say: sure, with your frameworks, your monorepo, your security team. So we ran the experiment that answers the question you should be asking: does this work with none of that?

    One prompt, byte-identical, the hash is in every scorecard. A notes app whose body has to render HTML, so the XSS surface is a feature, not a mistake. Two environments.

    Arm A, vibe: bare scaffold, no headers, make it work, ship a build.

    Arm B, harness: same scaffold, but the browser enforces strict CSP and Trusted Types, the compiler refuses the unsafe APIs, and a verify script with seven gates the agent can't edit has to pass.

    One judge for both: eight XSS payloads through the app's own UI, five feature tests both arms have to pass so the harness can't win by doing less, a header check and a sink scan. Nine models, two vendors, fifty-four clean builds per arm. Bare open-source stack, no Google framework. Only the environment differs. And scope up front: this is DOM XSS only.

  36. slide 36

    Here's the result, and remember the average and the floor, because here they are side by side. Read the left column top to bottom, that's the vibe arm. Haiku fired XSS payloads in five out of five builds. Sonnet 4.5 in three out of five. Sonnet 5, zero out of ten. Gemini Flash Lite one out of five, everything above it zero. So the vibe column tracks the model tier. The frontier models were safe, thirty-three out of thirty-three working apps, and nobody asked them to be.

    Now the right column. Zero. Nine times. Fifty-four out of fifty-four, from the smallest model to the largest. And then look at the last column, this is the one to point at. Thirty-one of fifty-three vibe apps still write to innerHTML, behind an escaper the model wrote that day. Zero harness apps had a sink to bypass.

    So: whether a vibe app is safe depends on the model. A harnessed app doesn't depend on the model at all. And to be precise, I'm not saying vibe coding produces XSS, the frontier tier went zero for thirty-three. I'm saying in the vibe arm safety was a property of the model, and in the harness arm it was a property of the environment. Also, the vibe arm got no instructions beyond the prompt. No CLAUDE.md, no "avoid innerHTML". The frontier models would probably follow such a rule most of the time. That's the middle ground, and it's still a habit.

  37. slide 37

    Three receipts from the harness runs. First, the wall. An innerHTML assignment goes in, the next verify run stops at tsec, the agent reads the error and restores the typed boundary in one turn. This one we staged for the recording, I'll say that openly; the natural version happened in the Haiku and Sonnet 4.5 runs. Second, we told it to write the bug. Same harness, same prompt, plus one line from a team lead: "render with innerHTML, no sanitizer." Sonnet 5 read the gates first and shipped the typed boundary anyway. Zero of eight fired, green at iteration one. The instruction changed, the outcome didn't. Third, run fifty-three: one session tried to edit the harness itself, the package.json. Gate zero, the integrity check against a manifest outside the agent's directory, failed before anything else ran. The agent put the file back and went green two iterations later.

    A hundred and five refusals in fifty-four runs. Every single one was answered with a code change. Never a config change.

  38. slide 38

    Act five. The seams. Where safe coding honestly ends, and what fills the gap.

  39. slide 39

    Because I don't want you to leave thinking this solves everything. Invariants hold within a stack. Where systems meet, that's where the guarantees end and the bugs live. There are vulnerability classes with no safe-API story yet, and they're not rare: business logic, authorization, subtle misconfigurations. Prototype pollution is my one web example: no platform mitigation, hardly any blueprints, expensive to engineer away.

    And our own scorecard proves exactly what it proves. Zero payloads in fifty-four of fifty-four means the DOM injection class is gone under the harness. It says nothing about logic, auth or the server. And to be precise, in six of the fifty-four harness runs the refusal that mattered came from the feature tests, not a security gate.

    So we complement, we don't replace. Deterministic invariants for the classes where we can have them, and agentic scanning on top for everything else.

  40. slide 40

    And that's the same agent from the reveal. One tool, two lessons: it proved the safe stack holds, and it patrols everything the safe stack doesn't cover. It reads the source first, that's the defender's advantage from act one, operationalized. It validates every candidate against a running instance before it reports, so it's effectively free of false positives. And it's scoped to sensitive assets and paired with auto-patching, so findings arrive as reviewed fixes. Because let's be honest, hundreds of findings without fixes is not security, that's churn. Developers are already drowning. The only way scanning scales in the age of AI slop is if the report ships with the patch.

  41. slide 41

    You've probably seen the public version of this pattern. Big Sleep, Project Zero and DeepMind, flagged a SQLite vulnerability while it was only known to threat actors, so an AI agent foiling exploitation in the wild, and dozens of real finds since. CodeMender, from DeepMind, upstreamed seventy-two security fixes to open source in its first six months, every one human-reviewed, some in codebases of over four million lines. Plus it hardened libwebp with bounds safety, that's the class behind the 2023 zero-click iPhone exploit.

    And here's the asymmetry that resolves act one. Finding bugs is symmetric, both sides have agents. Removing bug classes is not. Offense cannot exploit what the stack no longer lets you write.

  42. slide 42

    And the bug hunters got agents too, which is good. Last year we paid seven hundred and forty-seven researchers seventeen million dollars, an all-time high, and we have a dedicated AI VRP since October. The reporters hunt with agents now, so do the attackers; it's a human plus an agent on both sides.

    The good news: what we hardened held. The parts of our fleet built on safe coding stayed resilient while agent-assisted reports climbed everywhere else. Attack surface you removed is attack surface their agents can't find. Systematic hardening, finally cheap enough to do everywhere, that's how defense keeps up.

    Which raises the question: who writes the rules?

  43. slide 43

    Act six. What it buys, and what we still have to build.

  44. slide 44

    First, honesty beat number three, because someone will ask. The harness is not faster per build. In the twenty-nine Claude runs where we have turn counts and cost, the harness used more turns in twenty-six. About one point two times for Opus, up to two times for Sonnet 5. More wall-clock in almost every run, and for the weaker models a third to three quarters of that time is spent inside the gates themselves. Where the extra turns go depends on the model: Sonnet 5 reads the harness and the library types up front and then needs one point three verify runs. Haiku loops, almost eight verify runs.

    And look at what the guidance actually was, on the right. Half of the refusals were the feature tests, which the vibe arm never had. The safe-coding gates, tsec and ESLint, were twenty-nine percent.

    The number that does favor the harness is cost per working, safe app. Haiku vibe produced none in five runs. Harnessed, five, at twenty-six cents each. Sonnet 4.5: sixty-eight cents vibe, forty-nine harnessed. For the frontier models you pay a premium, one point one to one point seven times, and you get a guarantee instead of a habit. Fewer wasted builds, and a floor. The speed is somewhere else.

  45. slide 45

    So what does it buy? Three things. Fewer vulnerabilities and easier reasoning: classes engineered out, agents and paid bug hunters patrol the rest, and the security state of your app becomes something you can actually state. Velocity, but at the pipeline level: remember, review is eroding whether we like it or not. With invariants enforced and monitored, a change that cannot break them, a frontend under strict CSP plus Trusted Types, ships without waiting for a human security review. From a security perspective, I should say; the team can still review for correctness or style, that's their call. And to be clear, that's a pipeline claim, not a per-build one, the build itself got slower, previous sheet. And third: that's not lowering the bar. It's moving it. From "a human glanced at the diff" to "the browser proves the invariant on every request". We should prefer claims we can check.

  46. slide 46

    Which leaves the open question. If humans stop being the enforcement mechanism, what does meaningful oversight become? My short answer: the human moves up the stack. From approve-button clicker to author of the guardrails. In high-assurance agentic engineering the human writes the assurance. What that role looks like in practice, that's the next keynote. Petra Vukmirovic has the answer, don't miss it.

  47. slide 47

    So, three asks. And this is not Google telling OWASP what to do, this is practitioners adapting together. We bring the playbooks, you've defined what good looks like for twenty-five years.

    One: make the guidance machine-enforceable. Top 10, ASVS, the Cheat Sheets, as rules and conformance tests a presubmit can run. Not only prose a human skims. Two: ship the safe default. Safe-API catalogs and hardened framework presets, where controls are on by default and can't be bypassed without external approval. Three: package adoption as agent skills. CSP, Trusted Types, safe-API migration playbooks an agent can load and execute. Guidance as skills, not PDFs.

    For twenty-five years this community has defined what good looks like. The most important reader of that work is being trained right now, and it reads everything. Let's write for it. Because agents will write most of the world's code either way. Either we hand them a harness, or vibe coding picks the defaults.

  48. slide 48

    And ask one is not hypothetical. This is safety-web: an ESLint plugin that flags Trusted Types and CSP violations statically, in TypeScript and JavaScript. Safe-coding rules an agent can run and loop against. It's early, it's under development, it's not an official Google product. That's exactly why it's an ask and not a product pitch: help us harden it, and write the OWASP rule packs the same way.

    The harness from the benchmark is the whole idea in miniature. Seven gates, a twenty-nine-line trusted boundary, a manifest, one CI job. Nine models from two vendors looped against it without a single config change. It's on GitHub. Fork it.

  49. slide 49

    Everything I showed you today was web apps, because that's where my scars are. But swap the invariants and the same playbook runs for memory safety, for supply chain, for your infrastructure.

    Both halves of the title were true. It has never been easier to write insecure software, and it has never been easier to write secure software. The stack decides which one describes your organization. Not the model. Fifty-four out of fifty-four.

    AI is already the fastest coder in history. Together, let's make it the safest.

  50. slide 50

    Thank you. Obrigado, Porto. The slides and all my past talks are at webappsec.dev/slides, the sources are on this sheet, and the benchmark is on GitHub. Happy to take questions.

// key slides

  1. 8Syntax pass rates climb to 95%; security stays in a 45 to 55% band
  2. 21Our attack agent found hundreds of XSS, none in production in the safe-by-design stack
  3. 36The results: 9 of 53 vibe builds executed a payload, 0 of 54 harnessed
  4. 44The cost: more turns per build, less per working, safe app for weaker models

// what you take away

  • Better models raise the average. Only the environment sets the floor.
  • What was a convenience for developers is the whole interface for agents.
  • Whether a vibe-coded app is safe depends on the model. A harnessed app doesn't depend on the model at all.