mirrors/garage
Watch
1
0
Fork
You've already forked garage
 0
mirror of https://git.deuxfleurs.fr/Deuxfleurs/garage.git synced 2026-08-23 04:14:56 -04:00
Code Issues Releases Packages Wiki Activity
2,589 commits 31 branches 89 tags 51 MiB
Commit graph

2,589 commits

This branch
This branch
All branches
Author SHA1 Message Date
Alex
93eab8eaa3 Fixes to S3 compatibility page (#314) 
Mention PostObject is implemented, fix english mistakes

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/314
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-25 16:54:44 +02:00
Quentin Dufour
43ddc933f9
 		Update Ceph S3 endpoints compatibility 2022-05-25 15:20:08 +02:00
Alex Auvolat
9f303f6308
 		Shorter page title 2022-05-24 15:47:42 +02:00
Alex Auvolat
3be43f3372
 		Add lost content for Restic with Garage 
Suggested-by: Quentin <quentin@deuxfleurs.fr>
 2022-05-24 15:32:42 +02:00
Alex Auvolat
2da448b43f
 		Add documentation for new Admin API and a few infos on K2V 2022-05-24 15:28:37 +02:00
Alex
b2a2d3859f K2V client improvements (#307) v0.7.2 
- [x] Better distinguish error types
- [x] Parse error messages received from server
- [x] Remove `src/` folder layer, we don't have that for other crates

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/307
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-24 12:48:05 +02:00
Alex
382e74c798 First version of admin API (#298) 
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix #231
Fix #295

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-24 12:16:39 +02:00
trinity-1686a
64c193e3db Add a K2V client library and CLI (#303) 
lib.rs could use getting split in modules, but I'm not sure how exactly

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/303
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-05-18 22:24:09 +02:00
Alex Auvolat
c692f55d5c
 		K2V: Fix end parameter and add tests (fix #305) 2022-05-17 11:50:23 +02:00
Alex Auvolat
7b474855e3
 		Make background runner terminate correctly 2022-05-17 11:38:31 +02:00
Alex Auvolat
176715c5b2
 		Fix ReadIndex spec and add JSON5 remark to doc 2022-05-16 11:54:37 +02:00
Alex
5768bf3622 First implementation of K2V (#293) 
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-10 13:16:57 +02:00
Alex Auvolat
def78c5e6f
 		Update netapp to 0.4.4, fix #300 v0.7.1 2022-05-09 12:08:47 +02:00
Alex
277a20ec44 Fix layout show to not show changes when there are no changes (#297) 
fixes #295, partially

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/297
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-09 11:14:55 +02:00
Alex Auvolat
617f28bfa4
 		Correct small formatting issue 2022-05-05 14:21:57 +02:00
Mendes
948ff93cf1 Corrected the warnings and errors issued by cargo clippy 2022-05-01 16:05:39 +02:00
Alex Auvolat
3ba2c5b424
 		updated cargo.lock 2022-05-01 10:11:43 +02:00
Alex Auvolat
2aeaddd5e2
 		Apply cargo fmt 2022-05-01 09:57:05 +02:00
Alex Auvolat
c1d1646c4d
 		Change the way new layout assignations are computed. 
The function now computes an optimal assignation (with respect to partition size) that minimizes the distance to the former assignation, using flow algorithms.

This commit was written by Mendes Oulamara <mendes.oulamara@pm.me>
 2022-05-01 09:54:19 +02:00
Alex Auvolat
c9ef3e461b
 		fix clippy 2022-04-19 12:50:40 +02:00
Alex Auvolat
c93008d333
 		Prettier code for defragmentation 2022-04-19 12:50:40 +02:00
Alex Auvolat
e5341ca47b
 		Defragmentation in UploadPartCopy: first pass (not pretty but it compiles) 2022-04-19 12:50:36 +02:00
withinboredom
a4f9f19ac3 remove size limitation in UploadPartCopy (#280) 
This removes the >1mb s3_copy restriction.

This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248).

Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
 2022-04-19 12:49:43 +02:00
Baptiste Jonglez
47e57518ec Add documentation on running Kopia with Garage 2022-04-10 13:04:07 +02:00
Alex Auvolat
dffcd9f4b1
 		update Cargo.nix v0.7.0 2022-04-08 14:35:09 +02:00
Alex Auvolat
5d404dcd54
 		Add missing opentelemetry features 2022-04-08 14:21:04 +02:00
Quentin Dufour
62f0715abe Add/Fix OpenTelemetry 2022-04-07 16:12:35 +02:00
Quentin Dufour
7e1ac51b58 Add files to quickly test k8s 2022-04-07 16:12:35 +02:00
Alex Auvolat
94f1e48fff Update to netapp 0.4.2 (a tiny fix) 2022-04-07 11:50:03 +02:00
Alex Auvolat
cb5836d53c Bring maximum exponential backoff time down from 16h to 1h 2022-04-07 11:49:29 +02:00
Quentin Dufour
8e3ee82c3e Be clearer on what upgrades are (not) supported 2022-04-06 21:45:59 +02:00
Quentin Dufour
a122a8cb46 Add an "upgrading" section, add a guide for 0.7 2022-04-05 10:08:31 +02:00
Quentin Dufour
9fd8ec1dee Add documentation for winscp+sftpgo 2022-03-31 10:25:56 +02:00
Alex Auvolat
0091002ef2
 		New replication modes and their documentation 2022-03-28 16:26:04 +02:00
Alex Auvolat
8f9cf3a5d1
 		fix a clippy lint 2022-03-28 15:48:55 +02:00
Alex Auvolat
913f7754bb
 		Add blocks in errored state to garage stats 2022-03-28 15:47:23 +02:00
Alex Auvolat
42dde54126
 		Log admin GET requests at debug level instead of info 
to reduce noise in logs
 2022-03-28 15:46:52 +02:00
Alex Auvolat
dca2ffdf91
 		document administrative options 2022-03-28 12:26:08 +02:00
Quentin Dufour
0cf4efac89 Compile kuberetes-discovery only when release=true v0.7.0-rc1 2022-03-24 16:57:43 +01:00
Alex Auvolat
9d0ed78887 Add feature flag for Kubernetes discovery 2022-03-24 16:57:43 +01:00
Alex Auvolat
509d256c58
 		Make layout optimization work in relative terms 2022-03-24 15:27:14 +01:00
Alex Auvolat
2814d41842
 		Allow garage layout assign to assign to several nodes at once 2022-03-24 15:27:13 +01:00
Alex Auvolat
7e0e2ffda2
 		Slight change and add comment to layout assignation algo 2022-03-24 15:27:13 +01:00
Alex Auvolat
413ab0eaed
 		Small change to partition assignation algorithm 
This change helps ensure that nodes for each partition are spread
over all datacenters, a property that wasn't ensured previously
when going from a 2 DC deployment to a 3 DC deployment
 2022-03-24 15:27:10 +01:00
Alex Auvolat
43945234ae
 		Add missing src/block to toplevel cargo.toml 2022-03-23 10:26:10 +01:00
Alex Auvolat
3dc9214172
 		Add lots of comments on how the resync queue works 
(I don't really want to change/refactor that code though)
 2022-03-23 10:25:39 +01:00
Alex Auvolat
077dd1cde9
 		Clippy 2022-03-23 10:25:39 +01:00
Alex Auvolat
2d13f0aa13
 		run cargo2nix 2022-03-23 10:25:37 +01:00
Alex Auvolat
e480aaf338
 		Make background tranquility a configurable parameter 2022-03-23 10:25:19 +01:00
Alex Auvolat
8fd6745745
 		Move block RC code to separate rc.rs 2022-03-23 10:25:19 +01:00