NeroReflex/swfss3
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Feat: initial features implementation

Browse source
This commit is contained in:
benato.denis96@gmail.com 2026-03-12 08:45:23 +01:00
commit d3cae9fd20
5 changed files with 2613 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored Normal file
View file

@ -0,0 +1 @@
/target

1967
Cargo.lock generated Normal file
View file

File diff suppressed because it is too large Load diff

8
Cargo.toml Normal file
View file

@ -0,0 +1,8 @@
[package]
name = "swfss3"
version = "0.1.0"
edition = "2024"
[dependencies]
argh = "0.1.13"
s3 = { version = "0.1.22", features = ["blocking", "rustls"] }

208
README.md Normal file
View file

@ -0,0 +1,208 @@
# swfss3
`swfss3` is a small CLI for performing **filesystem-like operations** (list/read/write/delete)
against a **SeaweedFS S3-compatible** API endpoint using the Rust `s3` crate.
## Build
```bash
cargo build --release
```
Binary will be at `target/release/swfss3`.
## Authentication and configuration
You must provide:
- **Endpoint**: `--endpoint` (custom domain / S3 API URL)
- **Bucket**: `--bucket`
- **Region**: `--region` (defaults to `us-east-1`)
Authentication (choose one):
- **Flags**: `--access-key-id` and `--secret-access-key` (optional `--session-token`)
- **Environment variables**: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`
- **Anonymous**: `--anonymous` (unsigned requests)
If you see `invalid config: missing AWS_ACCESS_KEY_ID`, it means you did not pass credentials
flags and `swfss3` could not find the AWS environment variables. Fix it by using **one** of:
- **Provide keys via flags**:
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
--access-key-id YOUR_KEY --secret-access-key YOUR_SECRET \
list --prefix some/path/
```
- **Export keys via environment variables**:
```bash
export AWS_ACCESS_KEY_ID=YOUR_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET
# optional (temporary credentials)
export AWS_SESSION_TOKEN=YOUR_TOKEN
```
- **Use unsigned requests** (only works if the bucket/object is publicly readable/writable):
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style --anonymous \
list --prefix some/path/
```
Addressing style (optional, but often important for S3-compatible servers):
- `--path-style` (recommended for many self-hosted endpoints)
- `--virtual-hosted` (bucket as a subdomain)
- default is automatic selection
## Commands
All commands share the common flags above and then take their own arguments.
### List objects
List a prefix (like listing a directory):
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
list --prefix some/path/
```
Recursive listing (no delimiter grouping):
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
list --prefix some/path/ --recursive
```
Fetch all pages:
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
list --prefix some/path/ --all
```
### Read an object
To stdout:
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
read some/path/file.txt
```
To a local file:
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
read some/path/file.txt --out ./file.txt
```
### Write an object
Upload a local file to an object key:
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
write ./local.bin some/path/local.bin
```
Optionally set `Content-Type`:
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
write ./index.html some/path/index.html --content-type text/html
```
### Delete an object
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
delete some/path/local.bin
```
### Stat (HEAD) an object
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
stat some/path/file.txt
```
### Copy / move (rename) an object
Server-side copy:
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
cp some/path/file.txt some/path/file-copy.txt
```
Move (copy then delete):
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
mv some/path/file.txt some/path/renamed.txt
```
### Presigned URLs
Create a temporary download URL (default 900s expiry):
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
presign-get some/path/file.txt --expires 300
```
Create a temporary upload URL:
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
presign-put some/path/upload.bin --expires 300
```
### Delete everything under a prefix
Dry-run (prints keys that would be deleted):
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
rm-prefix some/path/
```
Actually delete:
```bash
swfss3 --endpoint http://localhost:8333 --bucket mybucket --path-style \
rm-prefix some/path/ --yes
```
## Troubleshooting
### `NoSuchBucket` but the bucket “exists”
If you get an error like `code=NoSuchBucket`, double-check:
- **You are hitting the S3 gateway endpoint**, not the filer UI or another reverse-proxied service.
- **The bucket exists in the same endpoint/cluster**. Buckets are endpoint-scoped.
- **Addressing style**: if one style doesn’t work, try the other:
- `--path-style` (common for self-hosted)
- `--virtual-hosted` (requires bucket DNS/wildcard)
## Help
```bash
swfss3 --help
swfss3 list --help
swfss3 read --help
swfss3 write --help
swfss3 delete --help
swfss3 stat --help
swfss3 cp --help
swfss3 mv --help
swfss3 presign-get --help
swfss3 presign-put --help
swfss3 rm-prefix --help
```

429
src/main.rs Normal file
View file

@ -0,0 +1,429 @@
use std::{
fs::File,
io::{self, BufWriter, Write},
path::PathBuf,
time::Duration,
};
use argh::FromArgs;
use s3::{AddressingStyle, Auth, BlockingClient, Credentials};
#[derive(FromArgs, Debug)]
/// Manage files in a SeaweedFS S3-compatible bucket.
struct Cli {
/// S3 endpoint URL, e.g. https://s3.example.com or http://localhost:8333
#[argh(option)]
endpoint: String,
/// signing region (SeaweedFS usually accepts "us-east-1")
#[argh(option, default = "String::from(\"us-east-1\")")]
region: String,
/// bucket name
#[argh(option)]
bucket: String,
/// AWS access key id (falls back to AWS_ACCESS_KEY_ID)
#[argh(option)]
access_key_id: Option<String>,
/// AWS secret access key (falls back to AWS_SECRET_ACCESS_KEY)
#[argh(option)]
secret_access_key: Option<String>,
/// AWS session token (falls back to AWS_SESSION_TOKEN)
#[argh(option)]
session_token: Option<String>,
/// do not sign requests
#[argh(switch)]
anonymous: bool,
/// always use path-style URLs (recommended for many S3-compatible servers)
#[argh(switch)]
path_style: bool,
/// always use virtual-hosted-style URLs (bucket as subdomain)
#[argh(switch)]
virtual_hosted: bool,
#[argh(subcommand)]
cmd: Command,
}
#[derive(FromArgs, Debug)]
#[argh(subcommand)]
enum Command {
List(CmdList),
Read(CmdRead),
Write(CmdWrite),
Delete(CmdDelete),
Stat(CmdStat),
Cp(CmdCp),
Mv(CmdMv),
PresignGet(CmdPresignGet),
PresignPut(CmdPresignPut),
RmPrefix(CmdRmPrefix),
}
#[derive(FromArgs, Debug)]
/// List objects under a prefix.
#[argh(subcommand, name = "list")]
struct CmdList {
/// key prefix to list (like a directory path)
#[argh(option, default = "String::new()")]
prefix: String,
/// list recursively (do not group by "/")
#[argh(switch)]
recursive: bool,
/// fetch and print all pages
#[argh(switch)]
all: bool,
/// maximum number of keys per page
#[argh(option)]
max_keys: Option<u32>,
}
#[derive(FromArgs, Debug)]
/// Read an object to a file (or stdout).
#[argh(subcommand, name = "read")]
struct CmdRead {
/// object key
#[argh(positional)]
key: String,
/// output file path (omit to write to stdout)
#[argh(option)]
out: Option<PathBuf>,
}
#[derive(FromArgs, Debug)]
/// Write a local file to an object key.
#[argh(subcommand, name = "write")]
struct CmdWrite {
/// local file path to upload
#[argh(positional)]
file: PathBuf,
/// destination object key
#[argh(positional)]
key: String,
/// optional content-type, e.g. text/plain
#[argh(option)]
content_type: Option<String>,
}
#[derive(FromArgs, Debug)]
/// Delete an object.
#[argh(subcommand, name = "delete")]
struct CmdDelete {
/// object key
#[argh(positional)]
key: String,
}
#[derive(FromArgs, Debug)]
/// Show object metadata (HEAD).
#[argh(subcommand, name = "stat")]
struct CmdStat {
/// object key
#[argh(positional)]
key: String,
}
#[derive(FromArgs, Debug)]
/// Copy an object to a new key (server-side copy).
#[argh(subcommand, name = "cp")]
struct CmdCp {
/// source key
#[argh(positional)]
src_key: String,
/// destination key
#[argh(positional)]
dst_key: String,
}
#[derive(FromArgs, Debug)]
/// Move/rename an object key (copy then delete).
#[argh(subcommand, name = "mv")]
struct CmdMv {
/// source key
#[argh(positional)]
src_key: String,
/// destination key
#[argh(positional)]
dst_key: String,
}
#[derive(FromArgs, Debug)]
/// Create a presigned download URL.
#[argh(subcommand, name = "presign-get")]
struct CmdPresignGet {
/// object key
#[argh(positional)]
key: String,
/// expiry in seconds
#[argh(option, default = "900")]
expires: u64,
}
#[derive(FromArgs, Debug)]
/// Create a presigned upload URL.
#[argh(subcommand, name = "presign-put")]
struct CmdPresignPut {
/// object key
#[argh(positional)]
key: String,
/// expiry in seconds
#[argh(option, default = "900")]
expires: u64,
}
#[derive(FromArgs, Debug)]
/// Delete all objects under a prefix (paged + batched deletes).
#[argh(subcommand, name = "rm-prefix")]
struct CmdRmPrefix {
/// key prefix to delete
#[argh(positional)]
prefix: String,
/// perform deletion (required); without this, prints keys that would be deleted
#[argh(switch)]
yes: bool,
}
fn main() {
let cli: Cli = argh::from_env();
if let Err(err) = run(cli) {
eprintln!("{err}");
std::process::exit(1);
}
}
fn run(cli: Cli) -> Result<(), s3::Error> {
if cli.path_style && cli.virtual_hosted {
return Err(s3::Error::invalid_config(
"choose only one of --path-style or --virtual-hosted",
));
}
let addressing_style = if cli.path_style {
AddressingStyle::Path
} else if cli.virtual_hosted {
AddressingStyle::VirtualHosted
} else {
AddressingStyle::Auto
};
let auth = if cli.anonymous {
Auth::Anonymous
} else if let (Some(akid), Some(secret)) = (cli.access_key_id, cli.secret_access_key) {
let mut creds = Credentials::new(akid, secret)?;
if let Some(token) = cli.session_token {
creds = creds.with_session_token(token)?;
}
Auth::Static(creds)
} else {
Auth::from_env()?
};
let client = BlockingClient::builder(&cli.endpoint)?
.region(cli.region)
.auth(auth)
.addressing_style(addressing_style)
.build()?;
match cli.cmd {
Command::List(cmd) => cmd_list(&client, &cli.bucket, cmd),
Command::Read(cmd) => cmd_read(&client, &cli.bucket, cmd),
Command::Write(cmd) => cmd_write(&client, &cli.bucket, cmd),
Command::Delete(cmd) => cmd_delete(&client, &cli.bucket, cmd),
Command::Stat(cmd) => cmd_stat(&client, &cli.bucket, cmd),
Command::Cp(cmd) => cmd_cp(&client, &cli.bucket, cmd),
Command::Mv(cmd) => cmd_mv(&client, &cli.bucket, cmd),
Command::PresignGet(cmd) => cmd_presign_get(&client, &cli.bucket, cmd),
Command::PresignPut(cmd) => cmd_presign_put(&client, &cli.bucket, cmd),
Command::RmPrefix(cmd) => cmd_rm_prefix(&client, &cli.bucket, cmd),
}
}
fn cmd_list(client: &BlockingClient, bucket: &str, cmd: CmdList) -> Result<(), s3::Error> {
let mut req = client.objects().list_v2(bucket);
if !cmd.prefix.is_empty() {
req = req.prefix(cmd.prefix);
}
if !cmd.recursive {
req = req.delimiter("/");
}
if let Some(max_keys) = cmd.max_keys {
req = req.max_keys(max_keys);
}
let print_page = |page: s3::types::ListObjectsV2Output| {
for p in page.common_prefixes {
println!("{p}");
}
for o in page.contents {
println!("{}", o.key);
}
};
if cmd.all {
for page in req.pager() {
print_page(page?);
}
} else {
print_page(req.send()?);
}
Ok(())
}
fn cmd_read(client: &BlockingClient, bucket: &str, cmd: CmdRead) -> Result<(), s3::Error> {
let obj = client.objects().get(bucket, cmd.key).send()?;
match cmd.out {
Some(path) => {
let file = File::create(path)
.map_err(|e| s3::Error::transport("failed to create output file", Some(Box::new(e))))?;
let mut writer = BufWriter::new(file);
obj.write_to(&mut writer)?;
writer
.flush()
.map_err(|e| s3::Error::transport("failed to flush output file", Some(Box::new(e))))?;
}
None => {
let stdout = io::stdout();
let mut handle = stdout.lock();
obj.write_to(&mut handle)?;
handle
.flush()
.map_err(|e| s3::Error::transport("failed to flush stdout", Some(Box::new(e))))?;
}
}
Ok(())
}
fn cmd_write(client: &BlockingClient, bucket: &str, cmd: CmdWrite) -> Result<(), s3::Error> {
let file = File::open(&cmd.file)
.map_err(|e| s3::Error::transport("failed to open input file", Some(Box::new(e))))?;
let len = file
.metadata()
.map(|m| m.len())
.map_err(|e| s3::Error::transport("failed to stat input file", Some(Box::new(e))))?;
let mut req = client
.objects()
.put(bucket, cmd.key)
.body_reader_sized(file, len);
if let Some(ct) = cmd.content_type {
req = req.content_type(ct);
}
req.send()?;
Ok(())
}
fn cmd_delete(client: &BlockingClient, bucket: &str, cmd: CmdDelete) -> Result<(), s3::Error> {
client.objects().delete(bucket, cmd.key).send()?;
Ok(())
}
fn cmd_stat(client: &BlockingClient, bucket: &str, cmd: CmdStat) -> Result<(), s3::Error> {
let out = client.objects().head(bucket, cmd.key).send()?;
if let Some(len) = out.content_length {
println!("content_length={len}");
} else {
println!("content_length=<unknown>");
}
if let Some(ct) = out.content_type {
println!("content_type={ct}");
}
if let Some(etag) = out.etag {
println!("etag={etag}");
}
Ok(())
}
fn cmd_cp(client: &BlockingClient, bucket: &str, cmd: CmdCp) -> Result<(), s3::Error> {
client
.objects()
.copy(bucket, cmd.src_key, bucket, cmd.dst_key)
.send()?;
Ok(())
}
fn cmd_mv(client: &BlockingClient, bucket: &str, cmd: CmdMv) -> Result<(), s3::Error> {
client
.objects()
.copy(bucket, cmd.src_key.clone(), bucket, cmd.dst_key)
.send()?;
client.objects().delete(bucket, cmd.src_key).send()?;
Ok(())
}
fn cmd_presign_get(
client: &BlockingClient,
bucket: &str,
cmd: CmdPresignGet,
) -> Result<(), s3::Error> {
let presigned = client
.objects()
.presign_get(bucket, cmd.key)
.expires_in(Duration::from_secs(cmd.expires))
.build()?;
println!("{}", presigned.url);
Ok(())
}
fn cmd_presign_put(
client: &BlockingClient,
bucket: &str,
cmd: CmdPresignPut,
) -> Result<(), s3::Error> {
let presigned = client
.objects()
.presign_put(bucket, cmd.key)
.expires_in(Duration::from_secs(cmd.expires))
.build()?;
println!("{}", presigned.url);
Ok(())
}
fn cmd_rm_prefix(client: &BlockingClient, bucket: &str, cmd: CmdRmPrefix) -> Result<(), s3::Error> {
let mut keys: Vec<String> = Vec::new();
for page in client.objects().list_v2(bucket).prefix(cmd.prefix).pager() {
let page = page?;
for obj in page.contents {
keys.push(obj.key);
}
}
if !cmd.yes {
for k in &keys {
println!("{k}");
}
eprintln!("dry run: {} objects (re-run with --yes to delete)", keys.len());
return Ok(());
}
// S3 multi-delete limit is 1000 keys per request.
for chunk in keys.chunks(1000) {
client
.objects()
.delete_objects(bucket)
.objects(chunk.iter().cloned())
.send()?;
}
Ok(())
}