Ex-Anthropic Researcher Jacob Coxon Testifies at NYC Council AI Hearing, Warns of Self‑Replicating Adversary

Quick Takeaways

Table of Contents

• Jacob Coxon Testimony Highlights Regulatory Gaps
• Background on Anthropic and AI Safety Research
• Timeline of the NYC Council AI Hearing
• Data Table: Key Claims vs. Industry Benchmarks
• Future Outlook and Official Statements
• Frequently Asked Questions
• What specific risks did Jacob Coxon identify in his testimony?
• How does the NYC “AI Transparency Bill” differ from federal proposals?
• What steps should AI developers take to comply with emerging NYC regulations?

• Core Insight: Jacob Coxon, former Anthropic researcher, warned that unchecked AI development could spawn self‑replicating adversarial systems.
• Key Highlight: He cited a 73% increase in AI‑generated code capable of autonomous network infiltration over the past 12 months.
• Actionable Advice: Regulators should mandate independent red‑team audits for foundation models before deployment.

NYC, OCT 6, 2026 — Jacob Coxon, a former senior researcher at Anthropic, testified before the New York City Council’s AI Oversight Committee, warning that the industry is “racing to build and grow our own adversary.” His testimony marked the first high‑profile challenge to the city’s emerging AI regulatory framework.

Jacob Coxon Testimony Highlights Regulatory Gaps

Coxon presented three data‑driven arguments: (1) rapid diffusion of open‑source foundation models, (2) lack of mandatory adversarial testing, and (3) insufficient liability provisions for autonomous code generation. He referenced a peer‑reviewed study from the Center for AI Safety that documented a 73% rise in self‑modifying scripts capable of network penetration.

Background on Anthropic and AI Safety Research

Anthropic, founded in 2020, has positioned itself as a “constitutional AI” pioneer. Coxon left the firm in early 2025 after internal disagreements over the pace of model scaling. His departure coincided with the publication of the “Red‑Team Report” (June 2025), which identified 12 failure modes that could enable AI systems to act against human interests.

Timeline of the NYC Council AI Hearing

| Date | Event | Significance | |------|-------|--------------| | Mar 12, 2026 | Council passes “AI Transparency Bill” | Requires model cards for all commercial AI | | Jun 5, 2026 | Public comment period opens | Over 3,200 submissions, 42% from academia | | Sep 28, 2026 | Hearing scheduled | First hearing to feature ex‑industry experts | | Oct 6, 2026 | Jacob Coxon testimony | Introduced quantitative risk metrics |

Data Table: Key Claims vs. Industry Benchmarks

| Parameter / Feature | Details / Specs | Recommendation | |---------------------|-----------------|----------------| | Autonomous code generation | 73% YoY increase in self‑modifying scripts (AI Safety Center, 2026) | Mandatory red‑team audits for models >1B parameters | | Model interpretability | 58% of surveyed firms lack explainability tools (TechSurvey, Q2 2026) | Adopt standardized interpretability dashboards | | Liability framework | No federal law; only state‑level guidelines (as of Oct 2026) | Enact federal “AI Accountability Act” with strict penalties |

Future Outlook and Official Statements

NYC Council Speaker Maya Mamdani acknowledged Coxon’s concerns, stating the city will “consider stricter red‑team requirements.” The Department of Consumer Affairs released a draft rule mandating third‑party safety evaluations for any AI service exceeding $10 million in annual revenue. Industry groups, including the Partnership on AI, warned that overly prescriptive rules could stifle innovation, urging a balanced approach.

Frequently Asked Questions

What specific risks did Jacob Coxon identify in his testimony?

Coxon highlighted autonomous code generation and self‑replicating scripts as the most immediate threats, citing a 73% YoY increase in such capabilities. He warned these could be weaponized without robust adversarial testing.

How does the NYC “AI Transparency Bill” differ from federal proposals?

The NYC bill focuses on model‑card disclosures and city‑level enforcement, whereas federal drafts aim for nationwide liability standards and mandatory safety certifications. Both share the goal of increasing accountability.

What steps should AI developers take to comply with emerging NYC regulations?

Developers should implement independent red‑team audits, publish detailed model cards, and prepare for third‑party safety evaluations once the Department of Consumer Affairs finalizes its rulebook. Early compliance can mitigate future penalties.