Architectural Decision Record (ADR) 001
Document Rule Processing System
Date: 2024-07-08, Status: Final
Context: Modernizing a legacy, file-system based Python/Node.js agent suite used for advanced document classification and automation within a Paperless-ngx environment by transitioning to an event-driven (Webhook) architecture.
Decision: Implement a stateful, buffered processing model with mandatory "Dry Run" validation before committing changes in the live production system.
1. Context and Problem Statement
The legacy system relied on local code injection (main.py reading Python logic) to execute rules against a Paperless instance. This tightly coupled design created high risks associated with untested rule deployment, potential database corruption from uncoordinated updates, and lack of an immediate safety net for changes (e.g., accidental tag removal).
The modernization goal is to decouple the Rule Editor from the core DMS while maintaining transactional integrity and providing a rigorous audit trail.
2. Decision Made: The Buffered Stateful Model
We will adopt a design where rules operate on an in-memory, cached representation of a document's state for the entire batch run. Changes (title modifications, tag additions, correspondent updates) are stored within this temporary in-memory buffer (@agents/temp/) and only committed to Paperless via a single bulk API update call at the end of the process.
Key Design Decisions & Rationale:
| Component | Legacy | New | Rationale |
|---|---|---|---|
| State Tracking | Immediate database modification per rule. | Buffered Changes, Only the final state is committed after all rules run. | Ensures atomicity within one execution cycle |
| Triggering | Direct code injection | Asynchronous Event | Decouples the Rule Editor logic from the DMS itself |
| Audit/State Check | Assumed state based on timestamps or file modifications | Audit Log, Every rule that modifies a document must write a structured metadata comment | Provides a clear, queryable audit trail. |
| Safety Net | Limited to pre-existing developer knowledge | Dry Run, A DryRun phase must run before any changes are committed to the production system. | Mitigates the risk of false positives. |
3. Rule Deployment Lifecycle
The deployment and modification of a rule follow a strict, multi-stage workflow, visualized below. The goal is to force continuous verification before live commitment.
graph TD
A[New / Edit Rule]
B{Dry Run}
C[Commit to Production]
D[Process ALL Documents for Consistency]
E(["System Operating State"])
A --> B
B -->|"Refinement"| A
B -->|"OK"| C
C --> D
D -->|"Problem"| A
D -->|"Stable"| E
classDef editor fill:#e6f7ff,stroke:#0099cc,stroke-width:2px;
classDef system fill:#f1f8e9,stroke:#5cb85c,stroke-width:2px;
class A,B,C,D editor;
class E system;
Links:
Appendix: Critical Decision Factors Summary
| Design Factor | Conclusion | Mitigation/Requirement |
|---|---|---|
| Event Handling | Shift from synchronous code injection to asynchronous webhook handling. | Requires building a robust queuing and state check mechanism (based on the Audit Log). |
| Process Safety | Changes must be buffered until all rules execute. | The system relies heavily on the final "Commit" step being an atomic transaction across all metadata fields. |
| Rule Creation | Rule creation is not just file generation; it's a multi-step process | The new editor/wizard must enforce this lifecycle, requiring multiple input stages before deployment approval. |