Databricks and Anthropic: The Five-Year AI partnership

Many organisations currently face challenges in developing and deploying AI agents that can effectively process enterprise data while meeting requirements for accuracy, security and access control.

Strategic partnerships have emerged as a popular way to face these challenges, between data infrastructure providers and AI model developers to bridge the gap between advanced AI capabilities and business-specific data requirements.

Now, Databricks and Anthropic have established a strategic five-year partnership to integrate Anthropic's language models (LMs) directly into the Databricks Data Intelligence Platform.

The agreement will make Anthropic's Claude models available to Databricks' customer base of more than 10,000 companies – focusing on enabling businesses to build AI agents capable of reasoning over their proprietary data.

Claude 3.7 Sonnet, described as a hybrid reasoning model designed for tasks including code generation, is now accessible through Databricks on major cloud computing services including AWS, Microsoft Azure and Google Cloud Platform.

Enterprise customers seek return on AI investments through Databricks' Mosaic AI

Databricks' Mosaic AI provides tools for building domain-specific AI agents based on an organisation's unique data.

These agents aim to deliver accurate results with governance across the data and AI lifecycle – and Anthropic's Claude models are designed to handle real-world tasks that customers require.

Therefore together, the companies will provide solutions for development, evaluation, deployment and governance of applications that use autonomous AI agents.

“As demand for data intelligence grows, our partnership with Anthropic allows enterprises to unlock the full potential of their data through AI,” says Ali Ghodsi, Co-founder and Chief Executive Officer of Databricks.

“We are bringing the power of Anthropic models directly to the Data Intelligence Platform – securely, efficiently and at scale – enabling businesses to build domain-specific AI agents tailored to their unique needs.”

Dario Amodei, Chief Executive Officer and Co-founder at Anthropic, says: “At Anthropic, we're watching AI transform businesses right now - not as some future promise.

“This year, we'll see remarkable advances in AI agents capable of working independently on complex tasks and with Claude now available on Databricks, customers can build even more powerful data-driven agents to stay ahead in this new era of AI.”

Databricks platform provides unified data with governance capabilities

The Databricks Data Intelligence Platform aims to democratise access to data and AI, making it easier for organisations to use their data for analytics, ML and AI applications.

The platform achieves this by being built on unified data and governance systems that help customers develop agents with domain knowledge drawn from their enterprise data.

Customers will use the platform for various applications including disease detection, climate change research, financial fraud detection, pharmaceutical development, mental health intervention and addressing financial inequality.

The benefits of Databricks and Anthropic’s partnership

The partnership provides several specific benefits.

Anthropic's Claude offers reasoning, planning and problem-solving capabilities that help customers build domain-specific AI agents capable of handling large, diverse datasets – and the large context window allows for better customisation.

The companies state that these capabilities can support complex multi-step workflows.

For instance, in healthcare, agents could improve the patient onboarding process for clinical trials.

Meanwhile, retail applications might include analysing sales data alongside seasonal patterns and inventory to recommend operational adjustments for staffing and store layouts.

Claude models will also integrate with the Databricks Data Intelligence Platform through SQL query and model endpoint capabilities.

This means organisations will not need to replicate data manually, which reduces costs and creates a more streamlined experience.

Enterprises can customise Claude models with retrieval augmented generation (RAG), a technique that enhances LMs by automatically generating vector indexes or fine-tuning models with enterprise data.

The role of data and AI governance 

The partnership also combines Anthropic's safety research and constitutional AI approach with Databricks' Unity Catalog, a unified governance solution for data and AI.

This way, customers gain governance across their data and AI assets and can enforce access controls, set rate limits to manage costs and track lineage throughout the AI workflow.

Then enterprises can implement safety guardrails, monitor for potential misuse and ensure AI systems operate within defined ethical boundaries while maintaining performance.

Jackie Brosamer, Vice President of Data and AI Platform Engineering at Block, says: “At Block, we emphasise practical, responsible and secure applications of AI.

“Through our strategic partnership with Databricks, we're able to leverage the newest and most advanced models, including Anthropic's Claude, both to support the development and serve as the underlying engine powering our internal deployment of codename goose, our open-source AI agent.”

Explore the latest edition of Technology Magazine and be part of the conversation at our global conference series, Tech & AI LIVE.

Discover all our upcoming events and secure your tickets today.

Technology Magazine is a BizClik bran