S3 destinations can often hit request limits when destinations are configured with high concurrent task and io limits. This usually results in common errors including 500 and 503.
# zcat /usr/local/jetapps/var/log/jetbackup5/s3/s3.log-20220712.gz
20220711T163900Z Message: Retry 1/3 upload file ... (HTTP code: 500): We encountered an internal error. Please retry the operation again later.
20220711T163901Z Message: Retry 1/3 upload file ... (HTTP code: 500): We encountered an internal error. Please retry the operation again later.
20220711T164411Z Message: Retry 1/3 upload file ... (HTTP code: 500): We encountered an internal error. Please retry the operation again later.
20220711T164426Z Message: Retry 1/3 upload file ... (HTTP code: 500): We encountered an internal error. Please retry the operation again later.
20220711T164426Z Message: Retry 1/3 upload file ... (HTTP code: 500): We encountered an internal error. Please retry the operation again later.
20220711T164426Z Message: Retry 1/3 upload file ... (HTTP code: 500): We encountered an internal error. Please retry the operation again later.
20220711T164426Z Message: Retry 1/3 upload file ... (HTTP code: 503): Storage service is temporarily not available. Please try again later (cause: Database:DBNoConnAvailable Detail=ConnWaitError at=db.(*DbAccess).Exec.func1( db/db_access.go:86 ))
20220711T164426Z Message: Retry 1/3 upload file ... (HTTP code: 503): Storage service is temporarily not available. Please try again later (cause: Database:DBNoConnAvailable Detail=ConnWaitError at=db.(*DbAccess).Exec.func1( db/db_access.go:86 ))
20220711T164847Z Message: Retry 1/3 upload file ... (HTTP code: 503): Storage service is temporarily not available. Please try again later (cause: Database:DBNoConnAvailable Detail=ConnWaitError at=db.(*DbAccess).Exec.func1( db/db_access.go:86 ))
20220711T164902Z Message: Retry 1/3 upload file ... (HTTP code: 503): Storage service is temporarily not available. Please try again later (cause: Database:DBNoConnAvailable Detail=ConnWaitError at=db.(*DbAccess).Exec.func1( db/db_access.go:86 ))
20220711T170044Z Message: Retry 1/3 upload file ... (HTTP code: 503): Storage service is temporarily not available. Please try again later (cause: Database:DBNoConnAvailable Detail=ConnWaitError at=db.(*DbAccess).Exec.func1( db/db_access.go:86 ))
20220711T170044Z Message: Retry 1/3 upload file ... (HTTP code: 503): Storage service is temporarily not available. Please try again later (cause: Database:DBNoConnAvailable Detail=ConnWaitError at=db.(*DbAccess).Exec.func1( db/db_access.go:86 ))
20220711T170045Z Message: Retry 1/3 upload file ... (HTTP code: 503): Storage service is temporarily not available. Please try again later (cause: Database:DBNoConnAvailable Detail=ConnWaitError at=db.(*DbAccess).Exec.func1( db/db_access.go:86 ))
20220711T170045Z Message: Retry 1/3 upload file ... (HTTP code: 503): Storage service is temporarily not available. Please try again later (cause: Database:DBNoConnAvailable Detail=ConnWaitError at=db.(*DbAccess).Exec.func1( db/db_access.go:86 ))
20220711T170045Z Message: Retry 1/3 upload file ... (HTTP code: 503): Storage service is temporarily not available. Please try again later (cause: Database:DBNoConnAvailable Detail=ConnWaitError at=db.(*DbAccess).Exec.func1( db/db_access.go:86 ))Fixes include
1. Reduce "Concurrent Backup Tasks" (Settings->Performance->Concurrent Restore/Download/Reindex Tasks OR Concurrent Backup/Clone Tasks)
2. Reduce IO Read/Write limits (Settings -> Resource -> IO Read Limit OR IO Write Limit)
3. Verify other servers or jobs are not backing up to the same destination simultaneously
