Apache Cassandra is renowned for its exceptional write performance, but even the best systems can be optimized further. Through careful tuning and strategic improvements, we achieved a remarkable 5x increase in write throughput in our production environment. This post shares the key strategies and configurations that made this dramatic improvement possible.
The Challenge
Our e-commerce platform was experiencing bottlenecks during peak traffic periods, with write operations becoming a significant constraint. The system was handling approximately 50,000 writes per second, but we needed to scale to 250,000+ writes per second to accommodate growing user demand and real-time analytics requirements.
Key Optimization Strategies
1. Batch Statement Optimization
Before: Individual INSERT statements
INSERT INTO user_events (user_id, event_type, timestamp, data)
VALUES (123, 'click', '2024-06-16 10:30:00', '{"page": "product"}');
INSERT INTO user_events (user_id, event_type, timestamp, data)
VALUES (124, 'view', '2024-06-16 10:30:01', '{"page": "home"}');
After: Optimized batch statements
BEGIN BATCH
INSERT INTO user_events (user_id, event_type, timestamp, data)
VALUES (123, 'click', '2024-06-16 10:30:00', '{"page": "product"}');
INSERT INTO user_events (user_id, event_type, timestamp, data)
VALUES (123, 'view', '2024-06-16 10:30:05', '{"page": "checkout"}');
INSERT INTO user_events (user_id, event_type, timestamp, data)
VALUES (123, 'purchase', '2024-06-16 10:30:10', '{"amount": 299.99}');
APPLY BATCH;
Key Principle: Batch statements targeting the same partition key to minimize coordinator overhead.
2. Write Path Configuration Tuning
Commitlog Settings:
# cassandra.yaml optimizations
commitlog_sync: batch
commitlog_sync_batch_window_in_ms: 2
commitlog_segment_size_in_mb: 64
commitlog_compression:
- class_name: LZ4Compressor
Memtable Configuration:
memtable_allocation_type: heap_buffers
memtable_heap_space_in_mb: 2048
memtable_offheap_space_in_mb: 2048
memtable_flush_writers: 4
3. Driver-Level Optimizations
Java Driver Configuration:
// Connection pooling optimization
PoolingOptions poolingOptions = new PoolingOptions()
.setConnectionsPerHost(HostDistance.LOCAL, 8, 32)
.setConnectionsPerHost(HostDistance.REMOTE, 2, 8)
.setMaxRequestsPerConnection(HostDistance.LOCAL, 1024)
.setMaxRequestsPerConnection(HostDistance.REMOTE, 256);
// Asynchronous writes with proper batching
Cluster cluster = Cluster.builder()
.addContactPoint("cassandra-node-1")
.withPoolingOptions(poolingOptions)
.withQueryOptions(new QueryOptions()
.setConsistencyLevel(ConsistencyLevel.LOCAL_ONE))
.build();
// Prepared statement for optimal performance
PreparedStatement preparedInsert = session.prepare(
"INSERT INTO user_events (user_id, event_type, timestamp, data) VALUES (?, ?, ?, ?)"
);
// Asynchronous execution with batching
List<ResultSetFuture> futures = new ArrayList<>();
for (Event event : eventBatch) {
BoundStatement boundStatement = preparedInsert.bind(
event.getUserId(),
event.getEventType(),
event.getTimestamp(),
event.getData()
);
futures.add(session.executeAsync(boundStatement));
}
4. Schema Design Improvements
Optimized Table Structure:
-- Before: Poor partition distribution
CREATE TABLE user_events_old (
event_id UUID PRIMARY KEY,
user_id INT,
event_type TEXT,
timestamp TIMESTAMP,
data TEXT
);
-- After: Optimized for write performance
CREATE TABLE user_events (
user_id INT,
event_date DATE,
event_time TIMESTAMP,
event_id UUID,
event_type TEXT,
data TEXT,
PRIMARY KEY ((user_id, event_date), event_time, event_id)
) WITH CLUSTERING ORDER BY (event_time DESC, event_id ASC)
AND compaction = {
'class': 'TimeWindowCompactionStrategy',
'compaction_window_unit': 'HOURS',
'compaction_window_size': 1
}
AND gc_grace_seconds = 864000;
5. Hardware and JVM Tuning
JVM Heap Configuration:
# Enhanced G1GC settings for write-heavy workloads
-Xms16G
-Xmx16G
-XX:+UseG1GC
-XX:G1HeapRegionSize=16m
-XX:MaxGCPauseMillis=200
-XX:+UnlockExperimentalVMOptions
-XX:+UseTransparentHugePages
-XX:+AlwaysPreTouch
System-Level Optimizations:
# Disk I/O optimization
echo noop > /sys/block/sda/queue/scheduler
echo 8 > /sys/block/sda/queue/read_ahead_kb
# Network buffer tuning
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
Results and Performance Metrics
Before Optimization
- Write Throughput: 50,000 operations/second
- Average Latency: 15ms (p95: 45ms)
- CPU Utilization: 75%
- Memory Usage: 85%
After Optimization
- Write Throughput: 250,000+ operations/second
- Average Latency: 3ms (p95: 8ms)
- CPU Utilization: 60%
- Memory Usage: 70%
Benchmark Results
Test Duration: 10 minutes
Data Size: 100M records
Metric Before After Improvement
Write Throughput 50K/sec 252K/sec 5.04x
P50 Latency 12ms 2.1ms 5.7x faster
P95 Latency 45ms 7.8ms 5.8x faster
P99 Latency 89ms 15.2ms 5.9x faster
CPU Efficiency +25% improvement
Memory Efficiency +18% improvement
Implementation Best Practices
1. Gradual Rollout Strategy
- Start with non-critical tables
- Monitor performance metrics continuously
- Implement circuit breakers for safety
- Use feature flags for quick rollback
2. Monitoring and Alerting
// Custom metrics for tracking performance
public class CassandraMetrics {
private final Timer writeLatency = Timer.build()
.name("cassandra_write_latency")
.help("Write operation latency")
.register();
private final Counter writeErrors = Counter.build()
.name("cassandra_write_errors")
.help("Write operation errors")
.register();
// Track batch sizes and success rates
private final Histogram batchSize = Histogram.build()
.name("cassandra_batch_size")
.help("Batch operation sizes")
.register();
}
3. Load Testing Framework
# Python load testing script
import asyncio
import time
from cassandra.cluster import Cluster
from cassandra.policies import DCAwareRoundRobinPolicy
async def write_load_test():
cluster = Cluster(
['cassandra-node-1', 'cassandra-node-2'],
load_balancing_policy=DCAwareRoundRobinPolicy()
)
session = cluster.connect('production_keyspace')
# Prepare statement for optimal performance
prepared = session.prepare("""
INSERT INTO user_events (user_id, event_date, event_time, event_id, event_type, data)
VALUES (?, ?, ?, ?, ?, ?)
""")
start_time = time.time()
operations = 0
for batch in generate_test_batches(1000): # 1000 batches
tasks = []
for event in batch:
task = session.execute_async(prepared, event)
tasks.append(task)
await asyncio.gather(*tasks)
operations += len(batch)
if operations % 10000 == 0:
elapsed = time.time() - start_time
throughput = operations / elapsed
print(f"Throughput: {throughput:.2f} ops/sec")
Troubleshooting Common Issues
Write Timeout Errors
# Increase timeout values for heavy write loads
write_request_timeout_in_ms: 10000
range_request_timeout_in_ms: 15000
Memory Pressure
# Monitor off-heap memory usage
nodetool info | grep "Off Heap Memory"
# Adjust memtable sizes if needed
memtable_cleanup_threshold: 0.2
Compaction Lag
-- Monitor compaction progress
SELECT * FROM system.compaction_history
WHERE keyspace_name = 'your_keyspace'
ORDER BY compacted_at DESC LIMIT 10;
Conclusion
Achieving a 5x write throughput improvement in Cassandra requires a holistic approach combining schema optimization, configuration tuning, driver improvements, and infrastructure enhancements. The key is to understand your specific workload patterns and systematically address bottlenecks at each layer of the stack.
Remember that performance optimization is an iterative process. Start with the highest-impact changes (usually batching and prepared statements), measure the results, and gradually implement additional optimizations while maintaining system stability.
The techniques outlined in this post have been battle-tested in production environments handling millions of writes per second. Adapt them to your specific use case, and always test thoroughly in a staging environment before deploying to production.
Leave a Reply