AmoebaFS
Developing Aspen in total isolation, absent a real-world use case to work out design and implementation flaws was obviously not a good strategy. AmoebaFS is being co-developed with Aspen to ensure the system is grounded in reality and can be used effectively.
The flexibility offered by Aspen’s design and implementation forms an almost ideal foundation for building a distributed file system that can be extended to support a broad range of features and that can continually alter its shape to adapt to changing runtime conditions.
Presently, it consists of a very simple and straight-forward implementation of a traditional POSIX file system that is exported to the outside world via the dCache NFS library. In the future though, it could be extended to support all kinds of features. To name a few:
- UnionFS style system. Read-only root w/ CopyOnWrite to writable layers
- Customizing the AllocationGroups used for each directory. Which would enable:
- Geo-location of directory content
- Per-directory control over baking media (NVMe vs 5400rpm hdd)
- All writes to NVMe w/ background migration to slower media
- Track hot/cold data and migrate between storage media
- Geo-dispursed base FS with per-site cache for hot files
- Background compression / deduplication
- Btrfs-style COW implementation with snapshotting support
- Embed a type UUID in the Inode to support multiple File implementations
- Could have implementation optimized for concurrent write access
- Optimizations for append-only log files
- Compressed files
The implementation has seen little actual testing and is nowhere near ready for actual use but the core functionality is in place.