mirrors/garage
Watch
1
0
Fork
You've already forked garage
 0
mirror of https://git.deuxfleurs.fr/Deuxfleurs/garage.git synced 2026-08-23 04:14:56 -04:00
Code Issues Releases Packages Wiki Activity
878 commits 31 branches 89 tags 51 MiB
Commit graph

878 commits

This branch
This branch
All branches
Author SHA1 Message Date
Quentin Dufour
8cd02639dc
 		drone: set TARGET env as needed by "to_s3" func 2022-08-03 11:19:26 +02:00
Alex Auvolat
e935861854
 		Factor out node request order selection logic & use in manager 2022-07-29 12:25:03 +02:00
Alex Auvolat
f0ee3056d3
 		Update cargo.nix 2022-07-29 12:25:03 +02:00
Alex Auvolat
126b037307
 		update netapp 2022-07-29 12:25:03 +02:00
Alex Auvolat
33750c04ed
 		Update cargo.nix 2022-07-29 12:25:03 +02:00
Alex Auvolat
68087ee13d
 		Fix clippy 2022-07-29 12:25:03 +02:00
Alex Auvolat
605a630333
 		Use streaming in block manager 2022-07-29 12:25:02 +02:00
Alex Auvolat
a35d4da721
 		update netapp to 0.5 2022-07-29 12:25:02 +02:00
Alex Auvolat
8e7e680afe
 		First adaptation to WIP netapp with streaming body 2022-07-29 12:25:02 +02:00
Alex Auvolat
16f6a1a65d
 		fix clippy 2022-07-29 12:24:49 +02:00
Alex Auvolat
ad35b18bb1
 		Faster chunker 2022-07-29 12:24:49 +02:00
Alex Auvolat
49154a78d8
 		Update cargo.nix 2022-07-29 12:24:48 +02:00
Alex Auvolat
ff4771c36a
 		cargo fmt 2022-07-29 12:24:48 +02:00
Alex Auvolat
381eb9a5a1
 		Fix tests 2022-07-29 12:24:48 +02:00
Alex Auvolat
2cad656a03
 		More make clippy happy 2022-07-29 12:24:48 +02:00
Alex Auvolat
0176da3ad2
 		Make clippy happy 2022-07-29 12:24:48 +02:00
Alex Auvolat
40150527b8
 		Update cargo.nix 2022-07-29 12:24:48 +02:00
Alex Auvolat
2f111e6b3d
 		Performance improvements: 
- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults
 2022-07-29 12:24:48 +02:00
Alex Auvolat
1b2e1296eb
 		Compute hashes on dedicated threads 2022-07-29 12:24:44 +02:00
Quentin Dufour
a184f0d0b5
 		Migrate to nix-daemon builders 2022-07-29 08:37:33 +02:00
Quentin Dufour
fcb04843f7
 		Run clippy in nix, leveraging nix caching ability 2022-07-26 18:27:52 +02:00
Quentin Dufour
5fb8584247
 		Refactor default.nix to follow Nix Flakes patterns 2022-07-26 18:27:52 +02:00
Quentin Dufour
96561c48a1
 		Bump Nix image to 22.05 2022-07-26 18:27:52 +02:00
Quentin Dufour
a49d0ea19f
 		Fix: compile aarch64+armv6 as static binaries 2022-07-26 18:27:51 +02:00
Quentin Dufour
9c9e483375
 		Put log-lines in nix.conf 2022-07-26 18:27:51 +02:00
Quentin Dufour
76cb34a0ae
 		Fail if compiled binary is dynamic 2022-07-26 18:27:46 +02:00
Alex Auvolat
ac03fa7937
 		Uniformize tracing::* imports (hopefully fixes 32-bit build) 2022-07-15 18:31:19 +02:00
Alex
4f38cadf6e Background task manager (#332) 
- [x] New background worker trait
- [x] Adapt all current workers to use new API
- [x] Command to list currently running workers, and whether they are active, idle, or dead
- [x] Error reporting
- Optimizations
  - [x] Merkle updater: several items per iteration
  - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on
- scrub:
  - [x] have only one worker with a channel to start/pause/cancel
  - [x] automatic scrub
  - [x] ability to view and change tranquility from CLI
  - [x] persistence of a few info
- [ ] Testing

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/332
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-08 13:30:26 +02:00
Alex Auvolat
aab34bfe54
 		add delays in k2v test_items_and_indices 2022-07-08 10:41:57 +02:00
Alex
fe3fa83de7 Publish k2v-client crate to crates.io (#337) 
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/337
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-04 18:27:25 +02:00
Alex Auvolat
b6d59ec19a
 		Fix poll item when item didn't change 2022-07-04 14:00:02 +02:00
Alex
0850bac874 Add poll command to k2v-cli (#335) 
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/335
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-04 12:45:32 +02:00
Alex
b74b533b7b Fix typo 2022-06-29 11:50:51 +02:00
Alex
996f2a6d58 Slides for talk at IMT Atlantique / STACK on 2022-06-23 (#333) 
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/333
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-23 14:28:40 +02:00
Alex
77e3fd6db2 improve internal item counter mechanisms and implement bucket quotas (#326) 
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/326
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-15 20:20:28 +02:00
Quentin Dufour
d544a0e0e0
 		Send CORS headers for all requests 2022-06-13 10:19:52 +02:00
Alex Auvolat
138e13071b
 		Fix garage_db build on 32-bit systems 2022-06-09 14:55:20 +02:00
Alex
b44d3fc796 Abstract database behind generic interface and implement alternative drivers (#322) 
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix #284

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-08 10:01:44 +02:00
Simon C
7eed3ceda9 docs: Add Trafik reverse proxy documentation 2022-06-07 16:16:52 +02:00
Simon C
4b8f48f3c5 docs: Fix title level 2022-06-07 13:32:52 +02:00
Simon C
7d3b5585f1 docs: Add link to facilitate navigation in the documentation 2022-06-07 09:38:59 +02:00
Quentin Dufour
a1abed0378
 		Remove useless MC_REGION env variable 2022-06-02 12:50:11 +02:00
Alex Auvolat
b54a938724 Fix garage_version() now that GIT_VERSION is read in crate garage_rpc v0.7.2.1 2022-06-02 12:00:10 +02:00
Alex Auvolat
ff06d3f082
 		Fix Content-Type headers for {admin,k2v} errors and admin responses 
Fix #315
 2022-05-25 17:09:33 +02:00
Alex
93eab8eaa3 Fixes to S3 compatibility page (#314) 
Mention PostObject is implemented, fix english mistakes

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/314
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-25 16:54:44 +02:00
Quentin Dufour
43ddc933f9
 		Update Ceph S3 endpoints compatibility 2022-05-25 15:20:08 +02:00
Alex Auvolat
9f303f6308
 		Shorter page title 2022-05-24 15:47:42 +02:00
Alex Auvolat
3be43f3372
 		Add lost content for Restic with Garage 
Suggested-by: Quentin <quentin@deuxfleurs.fr>
 2022-05-24 15:32:42 +02:00
Alex Auvolat
2da448b43f
 		Add documentation for new Admin API and a few infos on K2V 2022-05-24 15:28:37 +02:00
Alex
b2a2d3859f K2V client improvements (#307) v0.7.2 
- [x] Better distinguish error types
- [x] Parse error messages received from server
- [x] Remove `src/` folder layer, we don't have that for other crates

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/307
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-24 12:48:05 +02:00