writebackfs

A FUSE filesystem that sets up one directory as a writeback file cache for another.

This filesystem operates at file-level on top of existing filesystems and provides customizability for cache management policy through a dynamically loaded library.

The filesystem is designed for maximum data recoverability:

• All IO operations are handled atomically [1]
• All successful IO operations are guaranteed to have applied to the cache layer
• Data in cache layer are never evicted until successfully flushed to the base layer
• Defensive asserts terminate the system when in an undefined state to prevent unpredictable destructive damage

[1] Atomicity disclaimer

Atomicity is guaranteed so long as the FUSE process is running uninterrupted.
This includes IO errors from underlying filesystem as well as memory allocation failures.

However, there is no write-ahead log as in databases, so there is no additional protection against sudden process termination.

In practice, the scope of damage should be similar to using the underlying filesystems directly
- Crash mid-operation: Caller was waiting and operation is (potentially | partially) applied to cache layer
- Crash mid-flush: Data is still available in cache layer for recovery | Deletion target is not deleted

If FUSE terminated unexpectedly, manual intervention is necessary for recovering data and reconciling backing directories.

This filesystem is primarily targetted for file servers that wish to minimize / batch access to the base filesystem.
Examples include:

• NAS file server: Let HDD stay spun down by caching / buffering on SSD for low power
• Network folder: Reduce / batch network traffic

Requirements

Exclusive access to backing directories

This filesystem assumes complete ownership of the two underlying directories.
Therefore, they must not be altered outside of this filesystem or else changes may be lost.
This includes subdirectories and (sym)link targets.

While this filesystem is in use, treat the underlying directories as if they don't exist.

Similar projects

• Kernel space block-level caching
  • bcache: Require drive formatting before use
  • lvmcache: Require LVM, wrapper for dm-cache
  • dm-cache: Complex setup
  • flashcache: Unmaintained
• ZFS ARC: No generic writeback

Additionally, these options provide limited customizability in cache management policy,
and whether they also batch file creation / unlink / rename is uncertain.

Known issues

Rename flags are unsupported (Won't Fix)

Aside from renameat2() being non-standard, it was excluded due to the additional complexity expected when trying to retain atomicity guarantees.
May reconsider if circumstances change.

Mount failure (Out of scope)

FUSE may fail to mount if mount point is busy. This could happen if another process (eg: IDE) is watching the directory.

This is not a fault with the filesystem, but is included for completeness.

The default testing directory for end-to-end tests is set to a subdirectory within the project root,
which makes IDE users susceptible to the IDE's file watcher blocking filesystem mounts.

Add the testing directory to your IDE's file watcher ignore list or use a different testing directory

Feature incomplete (WIP)

This project is still under development and critical / complex features are prioritized over immediate usability to prevent being locked into architectural flaws.

The development plan is roughly as:

1. Essential IO operations are implemented
2. IO operations are batched correctly with atomic guarantees
3. Batched operations can be flushed correctly <<< Currently here (verifying with tests)
4. Custom cache management plugin can be loaded and share filesystem data <<< Partially implemented
5. Expose cache metrics
6. Refactor end-to-end test framework for mocking and flush complete alerts
7. Thread sleep / wake semantics on cache management works correctly
8. Allow configuration for opened files to be evictable
9. Implement passthrough IO
10. Non-essential IO operations are implemented
11. Filesystem can initialize with existing files in either of its underlying directories
12. Develop tool to assist base-cache reconciliation upon unexpected termination
13. Protect underlying directories upon initialization with empty read-only bind mounts
14. Write unit / integration tests for verifying filesystem internal states (See Testing#Test levels for its low priority)

---

Development

Dependencies

• Compiler with C++20 support
• Meson
• libfuse >= 3.18.2 (automatically built from source via Meson if unavailable)

Test dependencies

• fusermount3 executable binary in system $PATH

Compiling

0. cd <project root>
1. meson setup build
2. cd build
3. meson compile

Testing

0. cd <project root>
1. meson setup build
2. cd build
3. meson test <target> where target is:
   • e2e: End to end testing
     • --test-args=-a: Autoremove directories for successful testcases
     • --test-args='--include <regex>': Only run tests with matching names
     • --test-args='--exclude <regex>': Don't run tests with matching names
       • When both include and exclude are given, only run tests that match include but not exclude

Test levels

In the context of this project, the test levels are defined as:

• Unit test = Test internal functions and classes
• Integration test = Test IO handlers registered to FUSE
• End-to-end test = Test filesystem in use

Unit / Integration tests verify the internal state of the filesystem
for side-effects and pre/post-conditions, complementing asserts.

End-to-end tests verify the filesystem in use over multiple operations.

The most error-prone aspect of this project is cache coherency and deadlock prevention.
As these concerns surface mostly after multiple / concurrent operations,
end-to-end tests are given the highest importance among test levels.

Debugging

• All filesystem messages are logged via Syslog under the name writebackfs
  • View logs with journalctl -t writebackfs
    • -p [emerg|alert|crit|err|warning|notice|info|debug] for log level
• End to end test result includes the section number of its significant output
  • Indicates which section of the test scenario failed for what reason

Contributing

• All contributions must be thoroughly vetted and submitted by a human.
• The submitter is responsible for all of their contributions including:
  • Understanding of the contributions
  • Legal rights to the contributions
    • License / Patent / etc
  • Documentation / Explanation of the contributions
• Be mindful of the project's design priorities:
  1. Data recoverability
  2. Runtime stability
  3. Performance
  4. Minimize supply chain