extended attributes II

VFS and APFS

recap

If you read my previous entry on xattrs, you may already know that i am an idiot. I went through a small amount of trouble modifying grub to allow booting ext4 with a feature enabled that allows larger attributes.

VFS

Buuut, that only got me a few extra KiB. The VFS & xattr APIs still limit the size of all individual attributes, as well as the list of all attributes, to 64 KiB. That limit applies even if the filesystem itself supports larger attributes. This was a super bummer when I found out about it, especially finding out the dumb way, instead of reading ahead like i should have.

options

I went on a survey of various filesystems, and methods storing associated data: extended attributes versus named forks, NTFS, HFS+, APFS, BeFS, ... (Many years ago, i was working with xattrs in XFS, and could've swore this limit was not in place, but upon researching, i must be misremembering.) NTFS is suprisingly great for this. It has alternate data streams (ADS), that can store any arbitrary data for any path. The existing kernel drivers (the old r/o, and the newer r/w), as well as the new one being worked on right now (NTFSplus), do not support ADSs. NTFS-3g supports them beautifully, with "/path/to/file:stream", but is a FUSE-only driver, and (in my limited testing) doesn't handle permissions great for using as a dedicated FS. HFS+ is missing some features, and with BeFS you may as well just be running Haiku.

APFS

The new Apple filesystem, APFS, is great though! You can have a data and (single unnamed) resource fork, as well as arbitrarily large extended attributes. There is an APFS FUSE driver, that i had already been using, but, it's FUSE-only, as well as read-only. There is also linux-apfs-rw, a kernel driver, with write support, aaand everything in place internally for larger attributes and resource forks! But, linux xattr limits still stand in our way...

IOCTLs

Thankfully, the linux-apfs-rw driver is a fairly friendly codebase to work within. I was able to add new IOCTLs specifically for accessing extended attributes, and build a userspace library that uses them to mimic the xattr API, with limits removed/increased. (My branch is here, if you're curious. Though note it still needs cleanup, and isn't well tested yet.)

conclusion

After much filesystem fuckery, i've finally got an alternate data system that is actually usable. Now i can get back to working on my project!

<3 <3 <3

2025/DEC/02