Databricks Lakewatch to unleash agentic swarms on malware detect-and-destroy missions

Databricks is becoming an active malware hunter with Lakewatch - swarms of agents hunting down malware in customer-owned lakehouses storing telemetry data from IT, security and business data sources.

It says malware attackers are using AI agents to probe for security defence holes and insert malware into an enterprise’s IT estate at speeds that overwhelm human defenders. Enterprises and other organizations should combat this by using their own agent swarms to inspect security data from multiple sources ingested into a unified lakehouse. At the heart of this is the idea that diverse SIEM (Security Information and Event Management) data silos and their contents can be unified in an open format lakehouse where the full contents are available to AI agent swarms on malware detection and destroy missions. 

Ali Ghodsi, Co-Founder and CEO of Databricks, said: "Security teams can no longer rely on manual workflows to outpace AI-driven attacks. With Lakewatch, we are giving enterprises a new open data architecture and agentic capabilities to replace stagnating SIEM tools. Defenders must have even better visibility and speed than today’s agent attackers."

Lakewatch has a customer SIEM data lakehouse, using a customers’s own cloud storage, with Databricks providing the (agent) compute and analytics working on that data through its Unity Catalog - the same concept behind its core unstructured data lakehouse.

Structured and unstructured telemetry data from multiple sources is ingested into a unified, cloud-agnostic platform built on the Open Cybersecurity Schema Framework (OCSF). Data is stored in the Delta Lake or Apache Iceberg format. A Lakeflow Connect ETL facility handles ingestion and normalization of major security sources (AWS, Okta, Zscaler, etc) into standardized tables.

The needed agents are built with Databricks’ Agent Bricks facility and orchestrated with Databrick’s natural language Genie LLM. This, a Databricks blog says, "automates critical workflows such as ingesting and parsing new log sources to OCSF, authoring net-new detections based on the latest threat intelligence, modifying existing rules to reduce false positives, and translating natural language questions into SQL queries. Genie Spaces lets security teams query petabytes of data using plain English instead of specialized query languages, democratizing threat hunting across skill levels."

Databricks says these agents parse and enrich telemetry across hundreds of formats to reduce Mean Time to Detect & Respond (MTTD/R), while remaining inside a customer’s secure, governed environment.

It is setting up an Open Security Lakehouse Ecosystem of security vendors and delivery partners, including Akamai, Anvilogic, Arctic Wolf, Cribl, Obsidian, Okta, Palo Alto Networks, 1Password, Panther, Proofpoint, Rearc, Slack, TrendAI, Wiz (now part of Google Cloud), and Zscaler.

Lakewatch has automated testing and deployment to ensure defense is version-controlled and verified.

Adobe and Dropbox are cited as existing Lakewatch customers. AI company Anthropic is involved as Lakewatch  uses Claude models and their reasoning capabilities to correlate signals across security, IT, and business data to surface threats. Anthropic also uses Databricks for its own security lakehouse.

To help its Lakewatch initiative, Databricks is acquiring Antimatter and its software for secure authentication and authorization for AI agents. Antimatter was founded by UC Berkeley security researchers.

Databricks is also buying SiftD.ai for large-scale detection engineering and threat analytics technology. SiftG.ai was founded by the creator of Splunk’s Search Processing Language (SPL) and lead architects of Splunk's search stack. Terms for the Antimatter and SiftD.ai acquisitions were not disclosed.

Lakewatch is now available in Private Preview. Read more about it in a Databricks blog.