V2 Architecture

NanaSQLite's V2 mode is an optional architecture that provides non-blocking writes for write-heavy workloads.

Limitation

V2 mode is designed for single-process use. Multi-process setups such as Gunicorn with multiple workers are not supported.

Data Persistence Warning (SIGKILL)

V2 mode uses atexit to perform an automatic flush on shutdown, but if the process is killed by the OS (SIGKILL or power failure), buffered data in memory will be lost. For mission-critical data, call db.flush(wait=True) explicitly or use the default immediate mode.

Overview

The V2 engine uses a dual-lane design:

Application
  │
  ├─ KVS Lane ──────→ Staging Buffer → Commit
  │   (dict ops)       (write coalescing)
  │
  └─ Strict Lane ───→ Priority Queue → Ordered Execution
      (SQL ops)        (FIFO)

• KVS Lane: Processes dict-like operations such as db["key"] = value at high speed. Writes to the same key are coalesced
• Strict Lane: Executes SQL operations like sql_insert() and sql_update() in order

Enabling V2

from nanasqlite import NanaSQLite

db = NanaSQLite(
    "app.db",
    v2_mode=True,           # Enable V2 engine
    flush_mode="immediate",  # Flush mode
)

CRUD-Optimized Memory-First Mode

memory_first=True is a CRUD-optimized option built on the V2 engine's time-based flush. It loads the whole KVS table into memory at startup, then serves get / set / delete / len / keys / batch_get and similar CRUD operations from memory. Only changed keys are flushed back to SQLite in the background.

from nanasqlite import NanaSQLite

db = NanaSQLite(
    "app.db",
    memory_first=True,
    memory_flush_interval=5.0,  # Default: 5 seconds
)

db["session:1"] = {"user": "alice"}
assert db["session:1"]["user"] == "alice"

db.flush(wait=True)  # Explicit persistence checkpoint
db.close()

Regular v2_mode=True is for asynchronous writes. memory_first=True goes further: memory becomes the source of truth for CRUD operations, and SQLite is updated by periodic delta flushes. Use it only when the full dataset fits comfortably in memory and the application is single-process.

Caution

memory_first=True is not compatible with LRU / TTL / cache_size / cache_persistence_ttl. If the process is forcibly killed, changes since the last flush may be lost. Call db.flush(wait=True) at important boundaries.

Flush Modes

The V2 engine writes data to the database in the background. Choose from four flush modes:

Mode	Description	Data Safety	Performance
immediate	Flush after every operation	★★★★★	★★★
count	Flush after N operations accumulate	★★★★	★★★★
time	Flush at regular time intervals	★★★★	★★★★
manual	Manual flush only	★★★	★★★★★

immediate (Default)

The safest mode. Every operation is guaranteed to be written to the database:

db = NanaSQLite("app.db", v2_mode=True, flush_mode="immediate")
db["key"] = "value"  # Written to DB immediately

count

Flush after a specified number of operations accumulate:

db = NanaSQLite(
    "app.db",
    v2_mode=True,
    flush_mode="count",
    flush_count=100,  # Flush every 100 operations
)

time

Flush at specified intervals:

db = NanaSQLite(
    "app.db",
    v2_mode=True,
    flush_mode="time",
    flush_interval=3.0,  # Flush every 3 seconds
)

manual

Full manual control:

db = NanaSQLite("app.db", v2_mode=True, flush_mode="manual")

db["k1"] = "v1"
db["k2"] = "v2"
db["k3"] = "v3"

# Explicitly flush
db.flush()

Warning about manual mode

If the program exits without calling flush(), buffered data will be lost.

Chunk Size

For high-volume writes, adjust the chunk size to control transaction size:

db = NanaSQLite(
    "app.db",
    v2_mode=True,
    v2_chunk_size=1000,  # 1000 operations per transaction
)

Dead Letter Queue (DLQ)

Operations that fail to write are isolated in the Dead Letter Queue. This prevents errors from blocking subsequent operations.

The DLQ keeps up to 1000 entries by default. When the limit is reached, the oldest entry is evicted. Tune it with V2Config(max_dlq_size=...) or v2_max_dlq_size=....

Inspecting the DLQ

# Get DLQ contents
dlq = db.get_dlq()
for item in dlq:
    print(f"Failed operation: {item}")

Retrying the DLQ

# Retry failed operations
db.retry_dlq()

Clearing the DLQ

# Clear all entries in the DLQ
db.clear_dlq()

Metrics Collection

The V2 engine can collect detailed metrics such as processing latency, flush counts, and DLQ tracking. This is an opt-in feature.

db = NanaSQLite(
    "app.db",
    v2_mode=True,
    v2_enable_metrics=True,  # Enable metrics collection
)

# Get current metrics
metrics = db.get_v2_metrics()
print(f"Total Flushes: {metrics['total_flushes']}")
print(f"DLQ Count: {metrics['dlq_count']}")

StrictTask

You can enqueue custom tasks into the Strict lane:

from nanasqlite.v2_engine import StrictTask

# Enqueue a task with priority
task = StrictTask(
    priority=1,
    sequence_id=0,
    task_type="sql",
    sql="INSERT INTO logs (message) VALUES (?)",
    parameters=["Important event"],
    on_success=lambda: print("Success"),
    on_error=lambda e: print(f"Failed: {e}"),
)
db._v2_engine.enqueue_strict_task(task)

V2 Constructor Parameters

Parameter	Type	Default	Description
v2_mode	bool	False	Enable V2 engine
flush_mode	str	"immediate"	Flush mode
flush_interval	float	3.0	Interval for time mode (seconds)
flush_count	int	100	Threshold for count mode
v2_chunk_size	int	1000	Transaction chunk size
v2_max_dlq_size	`int	None`	1000
v2_enable_metrics	bool	False	Enable detailed metrics collection

Async V2 Usage

V2 mode also works with AsyncNanaSQLite:

from nanasqlite import AsyncNanaSQLite

async def main():
    db = AsyncNanaSQLite(
        "app.db",
        v2_mode=True,
        flush_mode="time",
        flush_interval=5.0,
    )

    await db.aset("key", "value")
    await db.aflush()

    dlq = await db.aget_dlq()

    db.close()

V1 vs V2 Comparison

Aspect	V1 (Default)	V2
Writes	Synchronous, blocking	Non-blocking (buffered)
Reads	Cache → DB	Cache → Buffer → DB
Data safety	Immediate guarantee	Depends on flush mode
Write performance	Baseline	High throughput
Complexity	Simple	Requires DLQ and flush management
Processes	Multi-process capable	Single-process only

When to Use V2

V2 is suitable for:

• Write-heavy applications (logging, sensor data, chat)
• Minimizing write latency
• Single-process applications
• memory_first=True when CRUD latency is the priority and the full dataset fits in memory

Stick with V1 when:

• Immediate data persistence is required
• Running in a multi-process environment (e.g., Gunicorn workers)
• Simple CRUD applications
• The full dataset cannot fit comfortably in memory