docs: add chunked uploads API documentation
Comprehensive documentation for the new chunked upload feature including: - API endpoint reference with parameters and responses - Usage examples in bash/curl and Python - Resumable upload examples - Parallel upload example - Configuration options and best practices
This commit is contained in:
parent
7e36d7d55c
commit
63acb5c81d
425
docs/content/doc/development/chunked-uploads.en-us.md
Normal file
425
docs/content/doc/development/chunked-uploads.en-us.md
Normal file
@ -0,0 +1,425 @@
|
||||
---
|
||||
date: "2024-01-08T00:00:00+00:00"
|
||||
title: "Chunked Uploads API"
|
||||
slug: "chunked-uploads"
|
||||
sidebar_position: 50
|
||||
toc: true
|
||||
draft: false
|
||||
menu:
|
||||
sidebar:
|
||||
parent: "development"
|
||||
name: "Chunked Uploads"
|
||||
identifier: "chunked-uploads"
|
||||
weight: 50
|
||||
---
|
||||
|
||||
# Chunked Uploads API
|
||||
|
||||
Gitea supports chunked uploads for large files, enabling resumable uploads that are resilient to network interruptions and timeouts. This is particularly useful for release attachments that may be hundreds of megabytes or larger.
|
||||
|
||||
## Overview
|
||||
|
||||
The chunked upload system works by:
|
||||
|
||||
1. Creating an upload session that tracks the upload state
|
||||
2. Uploading file chunks (in any order)
|
||||
3. Querying session status to resume interrupted uploads
|
||||
4. Completing the session to assemble chunks into the final attachment
|
||||
|
||||
## API Endpoints
|
||||
|
||||
All endpoints require authentication via token.
|
||||
|
||||
### Create Upload Session
|
||||
|
||||
Creates a new chunked upload session for a release attachment.
|
||||
|
||||
```
|
||||
POST /api/v1/repos/{owner}/{repo}/releases/{id}/assets/upload-session
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Name | Type | In | Description |
|
||||
|------|------|-----|-------------|
|
||||
| `owner` | string | path | Repository owner |
|
||||
| `repo` | string | path | Repository name |
|
||||
| `id` | integer | path | Release ID |
|
||||
| `name` | string | query | **Required.** Filename for the attachment |
|
||||
| `size` | integer | query | Total file size in bytes (recommended for validation) |
|
||||
| `chunk_size` | integer | query | Chunk size in bytes (default: 10MB, max: 100MB) |
|
||||
|
||||
**Response:** `201 Created`
|
||||
|
||||
```json
|
||||
{
|
||||
"uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
"file_name": "my-app-v1.0.0.zip",
|
||||
"file_size": 524288000,
|
||||
"chunk_size": 10485760,
|
||||
"chunks_expected": 50,
|
||||
"expires_at": 1704844800
|
||||
}
|
||||
```
|
||||
|
||||
### Upload Chunk
|
||||
|
||||
Uploads a single chunk of data to an existing session.
|
||||
|
||||
```
|
||||
PUT /api/v1/repos/{owner}/{repo}/uploads/{session_id}/chunks/{chunk_number}
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Name | Type | In | Description |
|
||||
|------|------|-----|-------------|
|
||||
| `owner` | string | path | Repository owner |
|
||||
| `repo` | string | path | Repository name |
|
||||
| `session_id` | string | path | Upload session UUID |
|
||||
| `chunk_number` | integer | path | Chunk number (0-indexed) |
|
||||
| Body | binary | body | Chunk data |
|
||||
|
||||
**Response:** `200 OK`
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_number": 0,
|
||||
"chunks_received": 1,
|
||||
"bytes_received": 10485760,
|
||||
"complete": false
|
||||
}
|
||||
```
|
||||
|
||||
### Get Upload Session Status
|
||||
|
||||
Returns the current status of an upload session. Use this to resume interrupted uploads.
|
||||
|
||||
```
|
||||
GET /api/v1/repos/{owner}/{repo}/uploads/{session_id}
|
||||
```
|
||||
|
||||
**Response:** `200 OK`
|
||||
|
||||
```json
|
||||
{
|
||||
"uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
"file_name": "my-app-v1.0.0.zip",
|
||||
"file_size": 524288000,
|
||||
"chunk_size": 10485760,
|
||||
"chunks_expected": 50,
|
||||
"chunks_received": 25,
|
||||
"bytes_received": 262144000,
|
||||
"received_chunks": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24],
|
||||
"status": "active",
|
||||
"expires_at": 1704844800
|
||||
}
|
||||
```
|
||||
|
||||
**Status Values:**
|
||||
|
||||
- `active` - Session is accepting chunks
|
||||
- `complete` - All chunks received, ready to finalize
|
||||
- `expired` - Session has expired (default: 24 hours)
|
||||
- `failed` - Upload failed
|
||||
|
||||
### Complete Upload Session
|
||||
|
||||
Assembles all uploaded chunks into the final attachment.
|
||||
|
||||
```
|
||||
POST /api/v1/repos/{owner}/{repo}/uploads/{session_id}/complete
|
||||
```
|
||||
|
||||
**Response:** `201 Created`
|
||||
|
||||
Returns the created attachment object (same format as regular attachment creation).
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 123,
|
||||
"name": "my-app-v1.0.0.zip",
|
||||
"size": 524288000,
|
||||
"download_count": 0,
|
||||
"created_at": "2024-01-08T12:00:00Z",
|
||||
"uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012",
|
||||
"browser_download_url": "https://gitea.example.com/attachments/b2c3d4e5-f6a7-8901-bcde-f23456789012"
|
||||
}
|
||||
```
|
||||
|
||||
### Cancel Upload Session
|
||||
|
||||
Cancels an upload session and deletes any uploaded chunks.
|
||||
|
||||
```
|
||||
DELETE /api/v1/repos/{owner}/{repo}/uploads/{session_id}
|
||||
```
|
||||
|
||||
**Response:** `204 No Content`
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Bash/curl
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
GITEA_URL="https://gitea.example.com"
|
||||
TOKEN="your_api_token"
|
||||
OWNER="myorg"
|
||||
REPO="myrepo"
|
||||
RELEASE_ID="1"
|
||||
FILE_PATH="/path/to/large-file.zip"
|
||||
CHUNK_SIZE=$((10 * 1024 * 1024)) # 10MB
|
||||
|
||||
FILE_NAME=$(basename "$FILE_PATH")
|
||||
FILE_SIZE=$(stat -f%z "$FILE_PATH" 2>/dev/null || stat -c%s "$FILE_PATH")
|
||||
|
||||
echo "Uploading $FILE_NAME ($FILE_SIZE bytes)"
|
||||
|
||||
# 1. Create upload session
|
||||
SESSION=$(curl -s -X POST \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
"$GITEA_URL/api/v1/repos/$OWNER/$REPO/releases/$RELEASE_ID/assets/upload-session?name=$FILE_NAME&size=$FILE_SIZE&chunk_size=$CHUNK_SIZE")
|
||||
|
||||
SESSION_ID=$(echo "$SESSION" | jq -r '.uuid')
|
||||
CHUNKS_EXPECTED=$(echo "$SESSION" | jq -r '.chunks_expected')
|
||||
|
||||
echo "Created session $SESSION_ID, expecting $CHUNKS_EXPECTED chunks"
|
||||
|
||||
# 2. Upload chunks
|
||||
CHUNK_NUM=0
|
||||
OFFSET=0
|
||||
while [ $OFFSET -lt $FILE_SIZE ]; do
|
||||
echo "Uploading chunk $CHUNK_NUM..."
|
||||
|
||||
dd if="$FILE_PATH" bs=$CHUNK_SIZE skip=$CHUNK_NUM count=1 2>/dev/null | \
|
||||
curl -s -X PUT \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
-H "Content-Type: application/octet-stream" \
|
||||
--data-binary @- \
|
||||
"$GITEA_URL/api/v1/repos/$OWNER/$REPO/uploads/$SESSION_ID/chunks/$CHUNK_NUM"
|
||||
|
||||
CHUNK_NUM=$((CHUNK_NUM + 1))
|
||||
OFFSET=$((CHUNK_NUM * CHUNK_SIZE))
|
||||
done
|
||||
|
||||
# 3. Complete upload
|
||||
ATTACHMENT=$(curl -s -X POST \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
"$GITEA_URL/api/v1/repos/$OWNER/$REPO/uploads/$SESSION_ID/complete")
|
||||
|
||||
echo "Upload complete!"
|
||||
echo "$ATTACHMENT" | jq .
|
||||
```
|
||||
|
||||
### Resuming an Interrupted Upload
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Resume a previously started upload
|
||||
|
||||
SESSION_ID="a1b2c3d4-e5f6-7890-abcd-ef1234567890"
|
||||
|
||||
# Get session status
|
||||
STATUS=$(curl -s \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
"$GITEA_URL/api/v1/repos/$OWNER/$REPO/uploads/$SESSION_ID")
|
||||
|
||||
# Get list of received chunks
|
||||
RECEIVED=$(echo "$STATUS" | jq -r '.received_chunks[]')
|
||||
CHUNKS_EXPECTED=$(echo "$STATUS" | jq -r '.chunks_expected')
|
||||
CHUNK_SIZE=$(echo "$STATUS" | jq -r '.chunk_size')
|
||||
|
||||
echo "Session has received chunks: $RECEIVED"
|
||||
echo "Expected chunks: $CHUNKS_EXPECTED"
|
||||
|
||||
# Upload missing chunks
|
||||
for ((i=0; i<CHUNKS_EXPECTED; i++)); do
|
||||
if ! echo "$RECEIVED" | grep -q "^$i$"; then
|
||||
echo "Uploading missing chunk $i..."
|
||||
dd if="$FILE_PATH" bs=$CHUNK_SIZE skip=$i count=1 2>/dev/null | \
|
||||
curl -s -X PUT \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
-H "Content-Type: application/octet-stream" \
|
||||
--data-binary @- \
|
||||
"$GITEA_URL/api/v1/repos/$OWNER/$REPO/uploads/$SESSION_ID/chunks/$i"
|
||||
fi
|
||||
done
|
||||
|
||||
# Complete
|
||||
curl -s -X POST \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
"$GITEA_URL/api/v1/repos/$OWNER/$REPO/uploads/$SESSION_ID/complete"
|
||||
```
|
||||
|
||||
### Python
|
||||
|
||||
```python
|
||||
import os
|
||||
import requests
|
||||
from pathlib import Path
|
||||
|
||||
def chunked_upload(gitea_url: str, token: str, owner: str, repo: str,
|
||||
release_id: int, file_path: str, chunk_size: int = 10*1024*1024):
|
||||
"""Upload a large file using chunked uploads."""
|
||||
|
||||
headers = {"Authorization": f"token {token}"}
|
||||
file_path = Path(file_path)
|
||||
file_size = file_path.stat().st_size
|
||||
|
||||
# Create session
|
||||
resp = requests.post(
|
||||
f"{gitea_url}/api/v1/repos/{owner}/{repo}/releases/{release_id}/assets/upload-session",
|
||||
headers=headers,
|
||||
params={
|
||||
"name": file_path.name,
|
||||
"size": file_size,
|
||||
"chunk_size": chunk_size
|
||||
}
|
||||
)
|
||||
resp.raise_for_status()
|
||||
session = resp.json()
|
||||
session_id = session["uuid"]
|
||||
|
||||
print(f"Created session {session_id}, expecting {session['chunks_expected']} chunks")
|
||||
|
||||
# Upload chunks
|
||||
with open(file_path, "rb") as f:
|
||||
chunk_num = 0
|
||||
while True:
|
||||
chunk = f.read(chunk_size)
|
||||
if not chunk:
|
||||
break
|
||||
|
||||
resp = requests.put(
|
||||
f"{gitea_url}/api/v1/repos/{owner}/{repo}/uploads/{session_id}/chunks/{chunk_num}",
|
||||
headers=headers,
|
||||
data=chunk
|
||||
)
|
||||
resp.raise_for_status()
|
||||
result = resp.json()
|
||||
print(f"Uploaded chunk {chunk_num}, total bytes: {result['bytes_received']}")
|
||||
chunk_num += 1
|
||||
|
||||
# Complete upload
|
||||
resp = requests.post(
|
||||
f"{gitea_url}/api/v1/repos/{owner}/{repo}/uploads/{session_id}/complete",
|
||||
headers=headers
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
return resp.json()
|
||||
|
||||
|
||||
def resume_upload(gitea_url: str, token: str, owner: str, repo: str,
|
||||
session_id: str, file_path: str):
|
||||
"""Resume an interrupted upload."""
|
||||
|
||||
headers = {"Authorization": f"token {token}"}
|
||||
|
||||
# Get session status
|
||||
resp = requests.get(
|
||||
f"{gitea_url}/api/v1/repos/{owner}/{repo}/uploads/{session_id}",
|
||||
headers=headers
|
||||
)
|
||||
resp.raise_for_status()
|
||||
session = resp.json()
|
||||
|
||||
received = set(session["received_chunks"])
|
||||
chunk_size = session["chunk_size"]
|
||||
chunks_expected = session["chunks_expected"]
|
||||
|
||||
print(f"Session has {len(received)}/{chunks_expected} chunks")
|
||||
|
||||
# Upload missing chunks
|
||||
with open(file_path, "rb") as f:
|
||||
for chunk_num in range(chunks_expected):
|
||||
if chunk_num in received:
|
||||
f.seek(chunk_size, 1) # Skip this chunk
|
||||
continue
|
||||
|
||||
chunk = f.read(chunk_size)
|
||||
resp = requests.put(
|
||||
f"{gitea_url}/api/v1/repos/{owner}/{repo}/uploads/{session_id}/chunks/{chunk_num}",
|
||||
headers=headers,
|
||||
data=chunk
|
||||
)
|
||||
resp.raise_for_status()
|
||||
print(f"Uploaded missing chunk {chunk_num}")
|
||||
|
||||
# Complete
|
||||
resp = requests.post(
|
||||
f"{gitea_url}/api/v1/repos/{owner}/{repo}/uploads/{session_id}/complete",
|
||||
headers=headers
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
return resp.json()
|
||||
```
|
||||
|
||||
### Parallel Uploads
|
||||
|
||||
Chunks can be uploaded in parallel for faster throughput:
|
||||
|
||||
```python
|
||||
import concurrent.futures
|
||||
|
||||
def upload_chunk(session_id, chunk_num, chunk_data):
|
||||
resp = requests.put(
|
||||
f"{gitea_url}/api/v1/repos/{owner}/{repo}/uploads/{session_id}/chunks/{chunk_num}",
|
||||
headers=headers,
|
||||
data=chunk_data
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return chunk_num
|
||||
|
||||
# Read all chunks
|
||||
chunks = []
|
||||
with open(file_path, "rb") as f:
|
||||
chunk_num = 0
|
||||
while True:
|
||||
chunk = f.read(chunk_size)
|
||||
if not chunk:
|
||||
break
|
||||
chunks.append((chunk_num, chunk))
|
||||
chunk_num += 1
|
||||
|
||||
# Upload in parallel (4 workers)
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
|
||||
futures = [
|
||||
executor.submit(upload_chunk, session_id, num, data)
|
||||
for num, data in chunks
|
||||
]
|
||||
for future in concurrent.futures.as_completed(futures):
|
||||
print(f"Completed chunk {future.result()}")
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Upload sessions have the following defaults:
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| Session expiry | 24 hours | Sessions expire after this time |
|
||||
| Default chunk size | 10 MB | Default size for each chunk |
|
||||
| Maximum chunk size | 100 MB | Maximum allowed chunk size |
|
||||
| Cleanup interval | 1 hour | How often expired sessions are cleaned up |
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Status Code | Description |
|
||||
|-------------|-------------|
|
||||
| `400 Bad Request` | Invalid parameters or chunk data |
|
||||
| `404 Not Found` | Session or release not found |
|
||||
| `410 Gone` | Session has expired |
|
||||
| `413 Request Entity Too Large` | File exceeds maximum attachment size |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always provide file size** - This enables chunk count validation and better progress tracking
|
||||
2. **Use appropriate chunk sizes** - Larger chunks (50-100MB) are more efficient for fast connections; smaller chunks (5-10MB) are better for unreliable networks
|
||||
3. **Implement retry logic** - Network errors on individual chunks should trigger retries, not full upload restarts
|
||||
4. **Query session status before resuming** - Always check which chunks were received before uploading more
|
||||
5. **Handle expiry gracefully** - If a session expires, create a new one and start over
|
||||
Loading…
Reference in New Issue
Block a user