Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

relay sync1.1 #961

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
1c1cc49
medsky radical trim of bigsky (relay)
brianolson Feb 12, 2025
702d73f
cleanup, add getRepo 302 redirect
brianolson Feb 21, 2025
3cdf190
more induction trace log
brianolson Feb 24, 2025
979c8ea
PR feedback and other cleanup
brianolson Feb 25, 2025
6ddbf48
note subscribeRepos message deprecation
brianolson Feb 26, 2025
664eccf
pass through #sync message
brianolson Feb 27, 2025
5553351
time-seq
brianolson Feb 28, 2025
52e19de
get echo log into same io.Writer as all other log
brianolson Feb 28, 2025
990d7b2
fix logging setup
brianolson Feb 28, 2025
72ad67b
fix RelaySetting gorm usage; TODO err rate squelch
brianolson Feb 28, 2025
dae5ceb
rename everything from 'bigsky' or 'medsky' to 'relay'
brianolson Feb 28, 2025
49c9d1c
GHA for relay
brianolson Feb 28, 2025
dbe624f
PR feedback
brianolson Mar 4, 2025
9053315
don't bounce auth key off db
brianolson Mar 4, 2025
71d35e4
handleFedEvent clean-er deprecated {handle,migrate,tombstone}
brianolson Mar 5, 2025
59f6718
a bunch of rename 'user' to 'account'
brianolson Mar 5, 2025
d445904
rename RepoManager to Validator
brianolson Mar 5, 2025
021d7cf
move validator into bgs package
brianolson Mar 5, 2025
b22b0b6
Validator is val
brianolson Mar 5, 2025
109e101
sync handler exists and checks a bunch of stuff
brianolson Mar 6, 2025
4048d23
rollback irrelevant changes
brianolson Mar 6, 2025
6ff02b6
err check, nil check
brianolson Mar 7, 2025
2018bd1
error object tweak
brianolson Mar 7, 2025
5f6a208
experimental tiny ram blockstore to see if it's better for small car …
brianolson Mar 7, 2025
59ae24f
TinyBlockstore as map
brianolson Mar 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .github/workflows/container-relay-aws.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: container-relay-aws
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should do a container-relay-ghcr CI action as well; we will remove the bigsky GHCR soon so it won't be a net increase in container builds

on: [push]
env:
REGISTRY: ${{ secrets.AWS_ECR_REGISTRY_USEAST2_PACKAGES_REGISTRY }}
USERNAME: ${{ secrets.AWS_ECR_REGISTRY_USEAST2_PACKAGES_USERNAME }}
PASSWORD: ${{ secrets.AWS_ECR_REGISTRY_USEAST2_PACKAGES_PASSWORD }}
# github.repository as <account>/<repo>
IMAGE_NAME: relay

jobs:
container-relay-aws:
if: github.repository == 'bluesky-social/indigo'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Setup Docker buildx
uses: docker/setup-buildx-action@v1

- name: Log into registry ${{ env.REGISTRY }}
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ env.USERNAME }}
password: ${{ env.PASSWORD }}

- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v4
with:
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=sha,enable=true,priority=100,prefix=,suffix=,format=long

- name: Build and push Docker image
id: build-and-push
uses: docker/build-push-action@v4
with:
context: .
file: ./cmd/relay/Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
271 changes: 271 additions & 0 deletions api/atproto/cbor_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions atproto/identity/cache_directory.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"github.com/hashicorp/golang-lru/v2/expirable"
)

// CacheDirectory is an implementation of identity.Directory with local cache of Handle and DID
type CacheDirectory struct {
Inner Directory
ErrTTL time.Duration
Expand Down
1 change: 1 addition & 0 deletions bgs/bgs.go
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ func NewBGS(db *gorm.DB, ix *indexer.Indexer, repoman *repomgr.RepoManager, evtm
slOpts.DefaultRepoLimit = config.DefaultRepoLimit
slOpts.ConcurrencyPerPDS = config.ConcurrencyPerPDS
slOpts.MaxQueuePerPDS = config.MaxQueuePerPDS
slOpts.Logger = bgs.log
s, err := NewSlurper(db, bgs.handleFedEvent, slOpts)
if err != nil {
return nil, err
Expand Down
5 changes: 5 additions & 0 deletions bgs/fedmgr.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ type Slurper struct {
shutdownResult chan []error

ssl bool

log *slog.Logger
}

type Limiters struct {
Expand All @@ -73,6 +75,7 @@ type SlurperOptions struct {
DefaultRepoLimit int64
ConcurrencyPerPDS int64
MaxQueuePerPDS int64
Logger *slog.Logger
}

func DefaultSlurperOptions() *SlurperOptions {
Expand All @@ -85,6 +88,7 @@ func DefaultSlurperOptions() *SlurperOptions {
DefaultRepoLimit: 100,
ConcurrencyPerPDS: 100,
MaxQueuePerPDS: 1_000,
Logger: slog.Default(),
}
}

Expand Down Expand Up @@ -115,6 +119,7 @@ func NewSlurper(db *gorm.DB, cb IndexCallback, opts *SlurperOptions) (*Slurper,
ssl: opts.SSL,
shutdownChan: make(chan bool),
shutdownResult: make(chan []error),
log: opts.Logger,
}
if err := s.loadConfig(); err != nil {
return nil, err
Expand Down
2 changes: 1 addition & 1 deletion cmd/bigsky/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,7 @@ func runBigsky(cctx *cli.Context) error {
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM)

_, err := cliutil.SetupSlog(cliutil.LogOptions{})
_, _, err := cliutil.SetupSlog(cliutil.LogOptions{})
if err != nil {
return err
}
Expand Down
Loading
Loading