An Introduction to Engula

Engula is a high-performance, memory-efficient database that is fully compatible with Redis. Built from Redis Server, Engula replaces Redis's in-memory object storage module with the Engula memory engine while maintaining complete protocol compatibility, enabling smooth migration from Redis.

Core Design Philosophy

Engula is designed with three core principles:

  • Full Redis Compatibility: Seamless migration with complete protocol compatibility
  • Memory Efficiency First: Data compression and metadata encoding to minimize memory footprint
  • Response Latency Priority: Sub-millisecond response times matching Redis performance

Architecture Overview

Engula Server is derived from Redis Server. The core transformation replaces Redis's in-memory object storage module with the Engula memory engine, while other Redis modules require only minimal adaptation, ensuring full compatibility with Redis.

Memory Architecture

In Redis Server, data access revolves around the Redis Object (robj). In Engula Server, when data is stored in memory, Redis Object is converted to Engula Object. Engula Object uses a more compact memory layout and is better adapted for compression/decompression operations.

When Redis performs KV operations, Engula Object is converted to a temporary Redis Object through:

  • Lazy Loading: Temporary robj objects are only created when actually used
  • Automatic Release: Using compiler mechanisms to ensure temporary robj objects are never left behind under any conditions

Memory Efficiency

Engula significantly reduces memory usage through optimized data structures and encoding:

Data Structure Redis (bytes) Engula (bytes) Optimization
HashTable 32 (dictEntry) 14 Swiss table
Key + robj 3 (key sds) + 16 (robj) 10 eobj storage
Value (String) 3 (value sds) 1 eobj storage
TTL 8 5 Variable-length integer encoding
Jemalloc ≥ 4 (8-byte alignment) 2 Memory Block management
Total 66 32 ~50% reduction

Memory Block Management

KV data is organized into Memory Blocks, which are divided into:

  • Mutable Zone: Data can be modified
  • Immutable Zone: Data can only be deleted

Background tasks compress and replace Memory Blocks in the Immutable Zone. These background tasks are split into small, interruptible and resumable execution units that run as coroutines embedded in native Redis's main thread and IO thread contexts.

The system automatically adjusts expected response times based on load conditions and schedules foreground tasks before deadlines. Background tasks are automatically distributed across different IO threads based on thread busyness to maximize CPU resource utilization.

Performance and Scalability

Horizontal Performance Scaling

Engula enables horizontal scaling by adding IO threads, with system throughput approaching or exceeding Redis. The architecture supports linear CPU scalability and predictable performance characteristics.

I/O Model Optimization

Redis: Busy Polling-based latency optimization model

  • Design philosophy: Assumes exclusive CPU and memory resources
  • I/O model: IO threads use "busy polling" mechanism, polling task queues one million times (approximately 0.1ms) before entering sleep
  • Performance characteristics: CPU usage shows non-linear growth under high load

Engula: Adaptive Signal/Wait efficiency optimization model

  • Design philosophy: Designed for efficient resource utilization in shared environments, minimizing unnecessary CPU waste
  • I/O model: Optimized Signal/Wait mechanism, completely removing high-consumption "busy polling"
  • Combines atomic operations with pthread condition variables to minimize thread wake-up (signal) call frequency and avoid redundant context switches
  • Performance characteristics: Achieves peak throughput comparable to Redis while maintaining more linear and predictable CPU scalability

CPU-for-Memory Trade-off

Engula implements real-time data compression/decompression to reduce data redundancy, trading CPU cycles for memory efficiency. This design significantly reduces memory usage while maintaining high performance.

Compatibility with Redis® / Valkey

Engula maintains full compatibility with Redis and Valkey protocols. By integrating the native compute layers of Open Source Redis and Valkey into its solution, Engula achieves protocol compatibility for data manipulation commands through careful identification of minimal storage integration points.

Engula can adapt and achieve protocol alignment within one month of new Redis or Valkey releases, ensuring seamless synchronization with the latest updates from both communities.

Version Support

  • Redis: Support up to version 7.2.4
  • Valkey: Full support for all versions
  • Transition: Actively transitioning to Valkey 8/9

Enhanced Features

Performance Improvements

  • Proprietary RDB Format: Significantly reduces Save/Load times and master-slave full synchronization time
  • Large Value Optimization: Optimized performance for large values, targeting 50%+ read performance improvement over Redis in multi-threaded (≥4 threads) model

Functionality Enhancements

  • ZSet Compression: Support for compressed sorted sets to further reduce memory usage
  • Enhanced Threading: Per-thread per-coroutine architecture with Redis/Valkey task priority for enhanced CPU resource utilization

Conclusion

Engula represents a significant evolution of Redis, delivering substantial memory savings through innovative memory management and compression techniques while maintaining full protocol compatibility. Its adaptive I/O model and horizontal scaling capabilities provide predictable, linear performance characteristics that surpass traditional Redis deployments.

Through memory-efficient design, performance optimizations, and complete Redis/Valkey compatibility, Engula offers a compelling solution for organizations seeking to reduce memory costs without sacrificing performance or migration complexity.