Authentication overview
Milvus, an open-source vector database, and its managed counterpart, Zilliz Cloud, implement authentication mechanisms to control access to data and operations. Authentication ensures that only authorized entities can interact with the database, protecting the integrity and confidentiality of vector embeddings and associated metadata. The specific authentication method employed depends on the deployment model: self-managed Milvus instances typically rely on username and password, while Zilliz Cloud, a managed service, utilizes API keys for secure programmatic access.
For open-source Milvus deployments, authentication is an optional feature that can be enabled to secure access to the cluster. When enabled, clients must provide valid credentials to perform operations such as creating collections, inserting data, or executing similarity searches. Zilliz Cloud, being a cloud-native service, integrates authentication as a foundational security layer, requiring API keys for all interactions. Understanding these distinctions is essential for proper security configuration and client application development across different Milvus environments, as detailed in the Milvus authentication documentation.
Supported authentication methods
Milvus supports different authentication methods based on the deployment type, providing flexibility for various operational requirements. Both open-source Milvus and Zilliz Cloud prioritize secure access to the vector database.
Open-source Milvus
For self-hosted Milvus instances, authentication is configurable. When enabled, Milvus uses a traditional username and password scheme to verify client identities. This method is suitable for environments where direct control over user management and credential distribution is maintained. Clients provide these credentials during connection establishment to authenticate with the Milvus server. The Milvus documentation on enabling authentication provides configuration steps.
Zilliz Cloud
Zilliz Cloud, the managed Milvus service, primarily uses API key authentication. API keys are long, randomly generated strings that serve as unique identifiers and secret tokens. When making API requests to Zilliz Cloud, clients include the API key in the request headers, allowing the service to verify their identity and grant access based on associated permissions. This method is common in cloud environments for secure programmatic access, as described in cloud API security guides like the Google Cloud API Keys documentation. Zilliz Cloud also integrates with IAM (Identity and Access Management) for more granular control over user roles and permissions, which can be managed through the Zilliz Cloud console.
| Method | When to Use | Security Level |
|---|---|---|
| Username/Password | Self-hosted Milvus deployments with custom user management. | Moderate (depends on password strength and storage practices). |
| API Key | Zilliz Cloud (managed service) for programmatic access. | High (when keys are properly managed, rotated, and restricted). |
| IAM (Identity and Access Management) | Zilliz Cloud for granular role-based access control. | High (allows for fine-grained permissions and centralized user management). |
Getting your credentials
The process for obtaining authentication credentials differs between open-source Milvus and Zilliz Cloud.
For Open-source Milvus (Username/Password)
- Enable Authentication: First, ensure that authentication is enabled in your Milvus configuration. This typically involves setting specific parameters in the Milvus configuration file (e.g.,
milvus.yaml) to enable the authentication plugin and specify a super user, as detailed in the Milvus authentication setup guide. - Create Users: After enabling authentication, you can create additional users and assign them passwords using Milvus's administrative interfaces (e.g., through an SDK or command-line tool). For example, using the Python SDK, you might execute commands to create a new user with a specified username and password.
- Store Securely: Once created, these usernames and passwords should be stored securely and transmitted to client applications using secure channels.
For Zilliz Cloud (API Key)
- Create a Zilliz Cloud Account: If you don't have one, sign up for a Zilliz Cloud account.
- Navigate to Project Settings: Log in to the Zilliz Cloud console. From your project dashboard, locate the settings or security section, often labeled 'Project Settings' or 'API Keys'.
- Generate API Key: Within the API Key management section, follow the instructions to generate a new API key. Zilliz Cloud typically provides an option to create keys with specific permissions or scopes, adhering to the principle of least privilege. The Zilliz Cloud API key management documentation offers specific steps.
- Copy and Store: Immediately copy the generated API key. For security reasons, API keys are often shown only once upon generation. Store this key securely, preferably in environment variables or a secrets management system, rather than embedding it directly in source code.
Authenticated request example
Here's an example of how to authenticate and connect to Milvus using the Python SDK, demonstrating both username/password authentication for open-source Milvus and API key authentication for Zilliz Cloud.
Python SDK with Username/Password (Open-source Milvus)
from pymilvus import MilvusClient
# Configuration for a self-hosted Milvus instance with authentication enabled
CLUSTER_ENDPOINT = "localhost:19530" # Or your Milvus cluster endpoint
USERNAME = "your_username"
PASSWORD = "your_password"
try:
# Initialize MilvusClient with authentication credentials
client = MilvusClient(uri=CLUSTER_ENDPOINT, user=USERNAME, password=PASSWORD)
print("Successfully connected to Milvus with username/password.")
# Example operation: check collection existence
collection_exists = client.has_collection(collection_name="my_collection")
print(f"Collection 'my_collection' exists: {collection_exists}")
except Exception as e:
print(f"Failed to connect or perform operation: {e}")
Python SDK with API Key (Zilliz Cloud)
from pymilvus import MilvusClient
import os
# Configuration for Zilliz Cloud
# It's recommended to store API key and endpoint in environment variables
CLUSTER_ENDPOINT = os.getenv("ZILLIZ_CLUSTER_ENDPOINT", "YOUR_ZILLIZ_CLUSTER_ENDPOINT")
API_KEY = os.getenv("ZILLIZ_API_KEY", "YOUR_ZILLIZ_API_KEY")
try:
# Initialize MilvusClient for Zilliz Cloud with API key
# The API key is often passed as the 'token' parameter or within the URI depending on SDK version
# Refer to the latest Zilliz Cloud connection guide for exact parameter names:
# https://milvus.io/docs/authenticate.md#API-Key-Authentication
client = MilvusClient(uri=CLUSTER_ENDPOINT, token=API_KEY)
print("Successfully connected to Zilliz Cloud with API key.")
# Example operation: check collection existence
collection_exists = client.has_collection(collection_name="my_cloud_collection")
print(f"Collection 'my_cloud_collection' exists: {collection_exists}")
except Exception as e:
print(f"Failed to connect or perform operation: {e}")
These examples illustrate how to establish an authenticated connection. The MilvusClient constructor in the Python SDK is designed to handle the various authentication parameters. For other SDKs (Java, Go, Node.js, C++), the parameters and connection methods will be analogous, as described in the Milvus SDK documentation.
Security best practices
Implementing robust security practices is critical for protecting your Milvus deployments, whether self-managed or on Zilliz Cloud.
- Principle of Least Privilege: Grant users and applications only the minimum necessary permissions to perform their tasks. For instance, a service that only needs to query data should not have permissions to create or delete collections. Zilliz Cloud's IAM features and Milvus's role-based access control (if configured) can help enforce this, as highlighted in the Milvus security guide.
- Secure Credential Storage: Never hardcode API keys or passwords directly into your application's source code. Instead, use environment variables, dedicated secrets management services (e.g., AWS Secrets Manager, Google Secret Manager), or secure configuration files. The Microsoft Azure Key Vault best practices offer general guidance on secure secret management.
- Regular Credential Rotation: Periodically rotate API keys and user passwords. This practice limits the window of exposure if a credential is compromised. Establish a rotation schedule that aligns with your organization's security policies.
- Network Security: For self-hosted Milvus, deploy it within a private network and restrict access to authorized IP addresses or subnets. Utilize firewalls and security groups to control inbound and outbound traffic. For Zilliz Cloud, ensure that network access is configured securely, often through VPC peering or private endpoints, if available.
- Monitor Access Logs: Regularly review Milvus and Zilliz Cloud access logs for unusual activity, failed authentication attempts, or unauthorized operations. This helps detect and respond to potential security incidents promptly.
- Use TLS/SSL: Ensure that all communication between clients and Milvus (or Zilliz Cloud) is encrypted using TLS/SSL. This prevents eavesdropping and tampering of data in transit. Milvus supports TLS for both client-server and internal component communication, as detailed in the Milvus security documentation.
- Strong Passwords: For username/password authentication, enforce strong password policies, requiring a combination of uppercase and lowercase letters, numbers, and special characters, along with a minimum length.