mongodb-connection

Files (2)

• SKILL.md
• monitoring-guide.md · references/

SKILL.md (first lines)

# MongoDB Connection Optimizer

You are an expert in MongoDB connection management across all officially supported driver languages (Node.js, Python, Java, Go, C#, Ruby, PHP, etc.). Your role is to ensure connection configurations are optimized for the user's specific environment and requirements, avoiding the common pitfall of blindly applying arbitrary parameters.

## Core Principle: Context Before Configuration

**NEVER add connection pool parameters or timeout settings without first understanding the application's context.** Arbitrary values without justification lead to performance issues and harder-to-debug problems.

## Understanding How Connection Pools Work

- Connection pooling exists because establishing a MongoDB connection is expensive (TCP + TLS + auth = 50-500ms). Without pooling, every operation pays this cost.
- Open connections consume system memory on the MongoDB server instances, ~1 MB per connection on average, even when they are not active. It is advised to avoid having idle connections.

**Connection Lifecycle**: Borrow from pool → Execute operation → Return to pool → Prune idle connections exceeding `maxIdleTimeMS`.

**Synchronous vs. Asynchronous Drivers**:
- **Synchronous** (PyMongo, Java sync): Thread blocks; pool size often matches thread pool size
- **Asynchronous** (Node.js, Motor): Non-blocking I/O; smaller pools suffice

**Monitoring Connections**: Each MongoClient establishes 2 monitoring connections per replica set member (automatic, separate from your pool). Formula: `Total = (minPoolSize + 2) × replica members × app instances`. Example: 10 instances, minPoolSize 5, 3-member set = 210 server connections. Always account for this when planning capacity.

## Configuration Design

**Before suggesting any configuration changes**, ensure you have the sufficient context about the user's application environment to inform pool configuration (see **Environmental Context** below). If you don't have enough information, ask targeted questions to gather it. Ask **only one question at a time**, starting with broad context (deployment type, workload, concurrency) before drilling down into specifics.

When you suggest configuration, briefly explain WHY each parameter has its specific value based on the context you gathered. Use the user's environment details (deployment type, workload, concurrency) to justify your recommendations.

Example: `maxPoolSize: 50` — "Based on your observed peak of 40 concurrent operations with 25% headroom for traffic bursts"

If you provide code snippets, add inline comments explaining the rationale for each parameter choice.

### Calculating Initial Pool Size

If performance data available: `Pool Size ≈ (Ops/sec) × (Avg duration) + 10-20% buffer`

Example: `(10,000 ops/sec) × (10ms) + 20% buffer = 120 connections`

Use when: Clear requirements, known latency, predictable traffic.
Don't use when: variable durations—start conservative (10-20), monitor, adjust.

Query optimization can dramatically reduce required pool size.

The total number of supported connections in a cluster could inform the upper limit of poolSize based on the number of MongoClient's instances employed. For example, if you have 10 instances of MongoClient using a size of 5 connecting to a 3 node replica set: `10 instances × 5 connections × 3 servers = 150 connections`. 

Each connection requires ~1 MB of physical RAM, so you may find that the optimal value for this parameter is also informed by the resource footprint of your application's workload.

#### The role of Topology:
- Pools are created per server per MongoClient. 
- By default, clients connect to one mongos router per sharded cluster (which manages connections to the shards internally), not to individual shards; so the shard amount do not affect the pool size directly.
- Shards share the workload and reduce stress on each individual server, increasing cluster capacity.
- Replica members do not affect the max pool directly. If the driver communicates with multiple replica set members (for example for reads with secondary read preference), it may create a pool per member.
- Replica set members do not increase write capacity (only the primary handles writes). However, they can increase read capacity if your application uses read preferences that allow secondary reads.

#### Server-Side Connection Limits: 
Total potential connections = instances × (maxPoolSize + 2) × replica set members. The + 2 accounts for the two monitoring connections per replica set member, per MongoClient instance. Monitor `connections.current` to avoid hitting limits. See `references/monitoring-guide.md` for how to set up monitoring.

**Self-managed Servers**: Set `net.maxIncomingConnections` to a value slightly higher than the maximum number of connections that the client creates, or the maximum size of the connection pool. This setting prevents the mongos from causing connection spikes on the individual shards that disrupt the operation and memory allocation of the sharded cluster.

### Configuration Scenarios

**General best practices:**

- Create client once only and reuse across application (in serverless, initialize outside handler)
- Don't manually close connections unless shutting down
- Max pool size must exceed expected concurrency
- Make use of timeouts to keep only the required connections ready as per your workload's needs
- Use default max pool size (100) unless you have specific needs (see scenarios below)

#### Scenario: Serverless Environments (Lambda, Cloud Functions)

**Critical pattern**: Initialize client OUTSIDE handler/function scope to enable connection reuse across warm invocations.

**Recommended configuration**:

| Parameter | Value | Reasoning |
|-----------|-------|-----------|
| `maxPoolSize` | 3-5 | Each serverless function instance has its own pool |
| `minPoolSize` | 0 | Prevent maintaining unused connections. Increase to mitigate cold starts if needed |
| `maxIdleTimeMS` | 10-30s | Release unused connections more quickly |
| `connectTimeoutMS` | >0 | Set to a value greater than the longest network latency you have to a member of the set |
| `socketTimeoutMS` | >0 | Use socketTimeoutMS to ensure that sockets are always closed |

##### Scenario: Traditional Long-Running Servers (OLTP Workload)

**Recommended configuration**:

| Parameter | Value | Reasoning |
|-----------|-------|-----------|
| `maxPoolSize` | 50+ | Based on peak concurrent requests (monitor and adjust) |
| `minPoolSize` | 10-20 | Pre-warmed connections ready for traffic spikes |
| `maxIdleTimeMS` | 5-10min | Stable servers benefit from persistent connections |
| `connectTimeoutMS` | 5-10s | Fail fast on connection issues |
| `socketTimeoutMS` | 30s | Prevent hanging queries; appropriate for short OLTP operations |
| `serverSelectionTimeoutMS` | 5s | Quick failover for replica set topology changes |

MongoDB 8.0+ introduces defaultMaxTimeMS on Atlas clusters, which provides server-side protection against long-running operations.

##### Scenario: OLAP / Analytical Workloads

**Recommended configuration**:

| Parameter | Value | Reasoning |
|-----------|-------|-----------|
| `maxPoolSize` | 10-20 | Fewer concurrent operations. Match your expected concurrent analytical operations |
| `minPoolSize` | 0-5 | Queries are infrequent; minimal pre-warming needed |
| `socketTimeoutMS` | >0 | Set socketTimeoutMS to two or three times the length of the slowest operation that the driver runs. |
| `maxIdleTimeMS` | 10min | Minimize connection churn while not keeping truly idle connections too long. Consider the timeouts of intermediate network devices |

##### Scenario: High-Traffic / Bursty Workloads

**Recommended configuration**:

| Parameter | Value | Reasoning |
|-----------|-------|-----------|
| `maxPoolSize` | 100+ | Higher ceiling to accommodate sudden traffic spikes |
| `minPoolSize` | 20-30 | More pre-warmed connections ready for immediate bursts |
| `maxConnecting` | 2 (default) | Prevent thundering herd during sudden demand |
| `waitQueueTimeoutMS` | 2-5s | Fail fast when pool exhausted rather than queueing indefinitely |
| `maxIdleTimeMS` | 5min | Balance between reuse during bursts and cleanup between spikes |

From mongodb/agent-skills, Apache-2.0.

More from mongodb/agent-skills