guardutils/filedust
2
0
Fork 0
Code Issues Pull Requests Actions Packages Releases 8 Wiki Activity
Files
main
filedust/DEVELOPMENT.md
T

639 lines
18 KiB
Markdown
Raw Permalink Blame History

# filedust Development Guide
Interested in the internals of filedust?
This guide describes the current `filedust` codebase for maintainers. It focuses on how the project is organised, how scanner rules flow into deletion candidates, and which safety invariants matter when changing the code.
---
## 1. What filedust does
`filedust` is a small command-line cleaner for obvious filesystem junk inside the invoking user's home directory.
Its core pipeline is:
```text
User-selected path
|
| filedust [PATH] [--dry-run] [-y]
v
Home-contained scanner
built-in junk rules
user include/exclude rules from ~/.filedust.conf
|
v
Findings list
path, kind, reason
|
v
Rich report and reclaimed-space estimate
|
| confirmation, unless --dry-run or --yes
v
Deletion of reported files and directories
```
`filedust` deliberately targets narrow, recognisable junk:
```text
Python and test caches
common build artifacts
editor backup files
temporary files
OS metadata files
user-configured include patterns
```
It is not a general disk-cleaning daemon, secure erase tool, duplicate finder, package manager, backup system, or sandbox.
---
## 2. Repository layout
The project is a single Python package under `src/filedust/`.
```text
src/filedust/
__init__.py package marker
cli.py argparse CLI, Rich output, confirmation, deletion
junk.py user rules, glob matching, filesystem scan
pyproject.toml Poetry package metadata and console script
poetry.lock locked dependency graph
README.md user-facing documentation
.filedust.conf.example example user include/exclude configuration
.pre-commit-config.yaml Bandit, Black, and generic pre-commit hooks
.gitea/workflows/ lint, dependency audit, SBOM, and Grype workflows
dist/ built release artifacts, not source
```
The installed command is configured in `pyproject.toml`:
```toml
[tool.poetry.scripts]
filedust = "filedust.cli:main"
```
There is no `filedust/__main__.py` at the time of writing, so `python -m filedust` is not the supported entry point. Use the installed `filedust` command or `poetry run filedust` during development.
---
## 3. Main runtime flow
All user-facing behaviour enters through `filedust.cli.main()`.
```text
filedust command
-> filedust.cli.main()
-> build argparse parser
-> install argcomplete hook
-> parse arguments
-> expand and resolve requested path
-> refuse paths outside $HOME
-> load ~/.filedust.conf rules
-> scan with junk.iter_junk()
-> compute approximate reclaimed size
-> render Rich report and summary
-> return on --dry-run
-> delete immediately on --yes
-> otherwise ask one confirmation prompt
-> delete reported findings
```
The command surface is intentionally small:
```text
PATH optional directory to scan, default: current directory
--dry-run show findings and summary without deleting anything
-y, --yes delete findings without the interactive confirmation prompt
--version print installed package version
```
Important dependency direction:
```text
cli.py
depends on junk.py, argparse, argcomplete, pathlib, shutil, Rich
junk.py
depends on pathlib, os.walk, configparser, fnmatch, dataclasses
```
Keep scanner and matching logic in `junk.py`. Keep CLI parsing, presentation, confirmation, and deletion orchestration in `cli.py`.
---
## 4. User rules
User rules are loaded from:
```text
~/.filedust.conf
```
The format is INI-like and uses no-value keys:
```ini
[exclude]
Projects/important/*
[include]
node_modules
*.tmp
```
`load_user_rules()` preserves key case by setting:
```python
parser.optionxform = str
```
Rule patterns are matched against paths relative to `$HOME`, not relative to the scan root.
Supported wildcard semantics are implemented in `matches_any()` and `_match_parts()`:
```text
* matches exactly one path segment
** matches zero or more path segments
```
Matching is case-sensitive because it delegates per-segment matching to `fnmatch()` without normalising case.
---
## 5. Built-in junk rules
Built-in junk directory names live in `junk.JUNK_DIR_NAMES`.
Current examples include:
```text
__pycache__
.pytest_cache
.mypy_cache
.ruff_cache
.nox
.tox
.hypothesis
.gradle
.parcel-cache
.turbo
.next
.vite
.sass-cache
dist
```
Built-in junk file patterns live in `junk.JUNK_FILE_PATTERNS`.
Current examples include:
```text
*~
*.swp
*.swo
*.swpx
*.tmp
*.temp
*.bak
*.orig
*.rej
.DS_Store
Thumbs.db
desktop.ini
```
Protected traversal directory names live in `junk.SKIP_DIR_NAMES`.
Current examples include:
```text
.cache
build
.gnupg
.git
.hg
.svn
.bzr
.idea
.vscode
```
These protected names are normally not descended into. If a user include pattern explicitly matches one of them, filedust reports the directory itself as a finding and still does not descend into it.
---
## 6. Finding objects
The scanner yields `junk.Finding` dataclass instances.
```python
@dataclass
class Finding:
path: Path
kind: str # "file" or "dir"
reason: str
```
Current `reason` values are:
```text
user_include
junk_dir
junk_file
broken_symlink
```
The CLI treats `kind == "file"` as an `unlink()` operation and `kind == "dir"` as a `shutil.rmtree()` operation. Keep this contract stable or update `delete_all()` with any new kinds.
---
## 7. Scanning behaviour
The scan entry point is `junk.iter_junk(root, rules=None)`.
Important scanner invariants:
```text
resolve the scan root before walking
walk with os.walk(..., followlinks=False)
match user rules relative to $HOME
apply user exclude before built-in junk detection
apply user include before built-in file junk detection
do not descend into symlink directories
do not follow valid symlink targets
handle unreadable paths without crashing
keep traversal contained in $HOME
```
The CLI performs the first home-boundary check before scanning:
```python
root_resolved.relative_to(Path.home().resolve())
```
`iter_junk()` also skips any walked directory that cannot be resolved relative to the module-level `HOME` constant.
### 7.1 Exclude rules
User excludes win over both user includes and built-in junk detection for a matching subtree or file.
For directories, an excluded path clears `dirnames` and prevents scanning that subtree:
```text
[exclude]
Projects/important/**
```
### 7.2 Include rules
User includes let the operator opt into paths that are not built-in junk.
Examples:
```text
[include]
node_modules
Downloads/**/*.tmp
```
Use include rules carefully. The scanner treats a matching directory as deletable and stops descending into it.
### 7.3 Symlinks
The scanner uses `lstat()` checks and `os.walk(..., followlinks=False)`.
Current behaviour:
```text
symlink directories are not descended into
valid symlink files are not auto-deleted by built-in rules
valid symlink files can be deleted only by user include rules
broken symlinks can be deleted by user include rules or by built-in junk file patterns that match the symlink name
symlink targets are not followed for scanning
```
Be careful when changing symlink behaviour. Any change must preserve the no-follow target boundary unless the tool's security model is deliberately redesigned.
---
## 8. Size reporting
The CLI computes an approximate reclaimed-space summary before deletion.
```text
file_size(path) -> path.stat().st_size, or 0 on error
dir_size(path) -> sum sizes of regular files under path.rglob("*")
compute_total_size() -> total over all findings
human_size() -> B, KB, MB, GB, TB, PB string
```
Size reporting is best-effort. It may differ from actual freed disk blocks because it uses file sizes rather than filesystem allocation data, because `stat()` can follow symlinks for explicitly included symlink file findings, and because the filesystem can change between scan, summary, and deletion.
---
## 9. Deletion behaviour
Deletion is centralised in `cli.delete_all(findings)`.
Current order:
```text
delete file findings first with Path.unlink(missing_ok=True)
delete directory findings second with shutil.rmtree(path)
print success or failure for each finding
return the number of failed deletions
```
The CLI calls `delete_all()` only after one of these conditions:
```text
--yes was provided
the user accepted the Rich confirmation prompt
```
`--dry-run` returns before deletion.
Do not add deletion paths outside `delete_all()` without preserving the same reporting and confirmation model.
---
## 10. Safety model
Important product boundaries:
```text
filedust only operates under the invoking user's $HOME
filedust does not automatically escalate privileges
filedust does not use sudo
filedust does not run as a daemon
filedust does not follow symlink targets while scanning
filedust previews all findings before deleting unless --yes is used
filedust deletes only paths yielded as findings
filedust does not securely wipe file contents
```
The most important guardrail is the home containment check in `cli.main()`. Preserve it for every operation that can scan or delete.
---
## 11. Development commands
Install dependencies:
```bash
poetry install
```
Run the CLI in the development environment:
```bash
poetry run filedust --help
poetry run filedust --dry-run .
```
Run pre-commit hooks:
```bash
poetry run pre-commit run --all-files
```
Run the pytest suite:
```bash
poetry run pytest
```
Current tests live under `tests/`. Scanner and rule coverage is in `tests/test_junk.py`; CLI, confirmation, and deletion coverage is in `tests/test_cli.py`. New behaviour should include focused tests, especially for `$HOME` containment, excludes, symlinks, dry-run, and deletion ordering.
Build release artifacts:
```bash
poetry build
```
---
## 12. Automation and security scanning
Gitea pull request workflow:
```text
.gitea/workflows/lint-and-security.yml
-> set up Python 3.13
-> install pre-commit
-> pre-commit run --all-files
-> install Poetry and poetry-plugin-export
-> export dependencies
-> pip-audit dependency audit
```
Scheduled/manual security workflow:
```text
.gitea/workflows/security-scan.yml
-> install verified Cosign, Syft, and Grype
-> generate SBOM
-> scan for vulnerabilities
-> notify Node-RED on fixable vulnerabilities
-> fail workflow on fixable Medium, High, or Critical vulnerabilities
```
Pre-commit currently includes Bandit, Black, trailing whitespace, EOF, YAML, and TOML checks.
Bandit is configured for `src/filedust/` and skips `B110` and `B112`, which reflects the codebase's deliberate best-effort exception handling in filesystem traversal.
---
## 13. Common maintenance tasks
### 13.1 Add a new CLI option
1. Add the argparse option in `cli.build_parser()`.
2. Decide whether it affects scanning, reporting, confirmation, or deletion.
3. Keep scanner changes in `junk.py` if filesystem enumeration or matching changes.
4. Keep output and prompt changes in `cli.py`.
5. Update README usage examples.
6. Add tests for parser behaviour and the affected operation.
### 13.2 Add a built-in junk pattern
1. Decide whether the pattern is safe enough to delete without project-specific context.
2. Add directory names to `JUNK_DIR_NAMES` or file patterns to `JUNK_FILE_PATTERNS`.
3. Consider whether the path should instead be a user include example.
4. Update README examples if it changes user-visible expectations.
5. Add tests covering the new rule and exclude precedence.
### 13.3 Change user rule matching
1. Start with `matches_any()` and `_match_parts()` in `junk.py`.
2. Preserve `$HOME`-relative matching unless intentionally breaking compatibility.
3. Preserve `*` and `**` semantics or document the compatibility break.
4. Preserve exclude precedence.
5. Update `.filedust.conf.example` and README.
6. Add tests for simple, recursive, and non-matching patterns.
### 13.4 Change deletion behaviour
1. Start with `delete_all()` in `cli.py`.
2. Keep scan/report and mutation separated.
3. Preserve `--dry-run` as non-mutating.
4. Preserve confirmation unless `--yes` is provided.
5. Preserve per-path failure reporting.
6. Consider filesystem races between scan and delete.
7. Add tests with temporary directories.
### 13.5 Add tests
Good first test areas:
```text
matches_any handles *, **, case sensitivity, and stripped slashes
load_user_rules reads include and exclude sections with case preserved
iter_junk detects built-in junk directories and files
iter_junk gives exclude rules precedence
iter_junk handles user includes
iter_junk does not follow symlink targets
CLI refuses paths outside $HOME
--dry-run does not call deletion
delete_all reports failures and deletes files before directories
```
Use temporary directories and temporary home directories where possible. Avoid tests that require root.
---
## 14. Important maintenance hazards
### 14.1 Home containment is the main safety boundary
`filedust` is marketed as safe because it refuses to operate outside `$HOME`. Any feature that accepts paths must preserve this boundary before scanning or deleting.
### 14.2 User includes are powerful
Built-in rules should stay conservative. User include rules intentionally allow broader cleanup. Keep the distinction clear in documentation and output.
### 14.3 Exclude precedence must remain predictable
Excludes are the operator's escape hatch. Avoid changes where an include or built-in rule can override an explicit exclude.
### 14.4 Symlink handling is security-sensitive
The scanner currently avoids following symlinks. Do not replace `lstat()` with `stat()` or enable `followlinks=True` without a deliberate redesign.
### 14.5 Filesystem races are unavoidable
The filesystem can change between scan, size calculation, confirmation, and deletion. Keep deletion best-effort and report failures clearly.
### 14.6 Directory names can be too broad
Names such as `dist` and `.next` are common build artifacts, but they can be meaningful in some projects. Be conservative when adding broad directory rules.
### 14.7 Size reporting is not an audit log
The reclaimed-space estimate is informational. Do not treat it as exact accounting or proof of deletion.
---
## 15. Troubleshooting guide
### 15.1 filedust refuses to run outside home
The requested scan root resolves outside `Path.home()`. Choose a path under the invoking user's home directory.
### 15.2 A path was not detected
Check, in order:
1. Is the path under `$HOME`?
2. Is the path under the selected scan root?
3. Is the path excluded by `~/.filedust.conf`?
4. Is the directory name in `SKIP_DIR_NAMES`?
5. Is it a valid symlink target that filedust intentionally does not follow?
6. Does the name match a built-in junk rule or user include rule?
### 15.3 A path was detected unexpectedly
Check whether it matches a built-in name or pattern, or whether a user include rule in `~/.filedust.conf` matches it relative to `$HOME`.
### 15.4 Deletion failed
The file may have been removed already, permissions may have changed, or the directory may no longer be removable. `delete_all()` reports each failed path and continues with the rest.
### 15.5 Completion does not work
Check that argcomplete is installed and registered for the shell:
```bash
eval "$(register-python-argcomplete filedust)"
```
---
## 16. Practical code-reading map
```text
Feature/question Start with Then read
CLI option behaviour cli.py:build_parser() cli.py:main()
Version output cli.py:get_version() pyproject.toml
Home boundary cli.py:main() junk.py:iter_junk()
Rich report cli.py:print_table() cli.py:print_summary_block()
Deletion cli.py:delete_all() junk.py:Finding
User config junk.py:load_user_rules() .filedust.conf.example
Pattern matching junk.py:matches_any() junk.py:_match_parts()
Built-in junk rules junk.py:JUNK_DIR_NAMES junk.py:JUNK_FILE_PATTERNS
Traversal skips junk.py:SKIP_DIR_NAMES junk.py:iter_junk()
Symlink behaviour junk.py:iter_junk() cli.py:delete_all()
Packaging pyproject.toml Poetry docs
Automation .gitea/workflows/ .pre-commit-config.yaml
```
---
## 17. Glossary
**Finding** A scanner result containing a path, kind, and reason.
**Junk directory** A directory whose basename is in `JUNK_DIR_NAMES`.
**Junk file** A file whose basename matches `JUNK_FILE_PATTERNS`.
**User include** A pattern in `~/.filedust.conf` that opts a path into cleanup.
**User exclude** A pattern in `~/.filedust.conf` that opts a path or subtree out of cleanup.
**Scan root** The user-selected path passed to `filedust`, defaulting to the current directory.
**Home boundary** The requirement that scan roots resolve under `Path.home()`.
**Dry run** A run that reports findings without deleting anything.
---
## 18. Final maintenance model
Most changes should preserve this model:
```text
Resolve the requested path under $HOME
-> scan without following symlink targets
-> apply user excludes before cleanup rules
-> report every deletion candidate before mutation
-> delete only after --yes or interactive confirmation
-> report failures without crashing the whole run
```
Before changing code, ask:
1. Does this preserve the `$HOME` containment boundary?
2. Does this preserve dry-run as non-mutating?
3. Does this preserve confirmation before deletion unless `--yes` is passed?
4. Does this preserve exclude precedence?
5. Does this avoid following symlink targets?
6. Is this built-in rule conservative enough, or should it be a user include example?
7. Are filesystem race behaviours clear and best-effort?
8. Do README examples, `.filedust.conf.example`, and shell completion still match the command surface?
9. Are there focused tests for the edge case being changed?
Keeping those boundaries clear is the main way to maintain filedust without turning a narrow junk cleaner into a risky general-purpose deletion tool.
Reference in New Issue View Git Blame Copy Permalink