Blixt Documentation v2.11

Start with the log: docker logs blixt or kubectl -n bfs logs deploy/<component>. Most startup problems are reported there in plain language.

The container exits immediately

/data is not a mounted volume. BlixtFS refuses to run with its data inside the container’s own filesystem, where it would be lost with the container. Add -v blixt-data:/data for a Docker volume, or -v /some/host/path:/data for a host directory.

The database directory “is still owned by 0:0 after chown”. /data is a host directory on a filesystem that ignores changes of owner, and the embedded database has to own its data directory. Use a Docker volume: -v blixt-data:/data. See Known issues.

The configuration is invalid. The log names the setting. Check YAML indentation, and that bucket entries sit under the right provider.

The auth token was refused. The log says “refused this process’s auth token (AUTH_TOKEN)”. Every BlixtFS server in a deployment needs the same AUTH_TOKEN.

The write tier is off in a scale-out deployment. The log says “a scaleout deployment requires the write tier”. Remove write.enabled: false or --write=false.

A bucket doesn’t appear

  • Check the log for the bucket’s name. Credential and permission errors are reported per bucket.
  • Check that the bucket is under the right provider directory: an S3 bucket appears in aws/, a GCS bucket in gcp/.
  • For S3, check that the bucket’s region is correct.
  • Large buckets may still be indexing. Mark them lazy to serve them at once.

Mounting fails

Connection refused or timed out. Check that the port is published (-p 2049:2049) and not blocked by a firewall. For a Compose deployment, remember the example binds ports to 127.0.0.1.

MacOS: “Operation not permitted”. Add noresvport to the mount options.

SMB on MacOS: port 445 is in use. MacOS File Sharing already uses port 445. Use NFS instead, or turn off File Sharing.

NFSv3 doesn’t work. NFSv3 needs a license and, in Docker, host networking. Use NFSv4 otherwise.

Changes made in the bucket don’t show up

  1. Check that notifications are working. The log reports whether notifications were set up for each bucket, and why not if they weren’t. For Google Cloud Storage, the storage service agent needs the roles/pubsub.publisher role. On any directory in the bucket, getfattr -n bfs.notifications says when an event was last applied and how many are waiting:

    bfs.notifications="last applied 12s ago; 2 waiting, the oldest for 40s"
    

    Events waiting a long time mean they are arriving faster than they are applied, or the queue is shared with other traffic. Start BlixtFS with --log_pubsub to log a line for every event applied, saying how long after the event it took effect, and every 30 seconds while events are waiting. Those lines start with pubsub:.

  2. Check the provider. CoreWeave has no notifications, and Cloudflare R2 needs an API token for them. For these, changes arrive with the next consistency check or rescan.

  3. Check the client’s cache. NFS clients cache attributes and directory lookups. See Connecting clients for mount options that reduce caching.

  4. Ask for a check. setfattr -n bfs.fsck_request -v 1 <directory> brings that directory up to date. See Consistency checks.

File operations fail with I/O errors

The cloud provider is refusing requests. The Cloud dashboard and the log show provider errors: expired credentials, missing permissions or quota limits.

Local disk is full. Once disk use reaches cache.stop_new_files_percent, no new files are written, and the log says “Refusing new files”. Files already being written can finish. Once it reaches cache.stop_write_percent, every write is refused, and the log says “Refusing writes”. Either way, clients see “No space left on device” until disk space is freed up. Check upload errors in the log, and give /data more space.

With write servers, a full write server gets no new files. A file whose write server fills up while it is being written moves to a write server with room for all of it, and the write goes on there. Clients see “No space left on device” only when too few write servers have room.

Getting help

Collect the BlixtFS version (from the startup banner), your configuration with secrets removed, and the log around the time of the problem. Then contact us.