Skip to main content
The Amazon S3 connector points Bag of words at a bucket, optionally scoped to a key prefix. The agent lists objects, greps them, and reads them — including inside PDF, Word, PowerPoint, Excel and CSV — and can attach an object to a report. Objects too large to load whole can be read in byte-range windows.
This connector is in beta. Fields and behavior may change between releases.
S3 appears under the database category in the add-connection grid, but it is a file connection: the agent sees files and reads them, it does not write SQL against them.

​
How it works

Bag of words builds a file catalog from the bucket. It uses the same parsing and search pipeline as Files and Directories, with object-store-appropriate limits:
  • list_files — a bounded ListObjectsV2 under the prefix.
  • read_file — GetObject, with .csv/.tsv/.xlsx/.xls parsed into tables and .pdf/.docx/.pptx text-extracted.
  • Windowed reads — a byte-range GetObject for objects too large to load whole (logs, ndjson, big CSVs).
  • grep_files — line-level grep with an explicit file, byte and time budget, and a resumable cursor.
There is no live full-content search: a content scan over an object store is one GetObject per key, so keyword search runs against the indexed catalog instead. Writes are not supported. Everything is confined to bucket + prefix. A file id is the key relative to the prefix, and any id that escapes it is rejected.

​
Before you start

  • The bucket exists, and you know its region (or the endpoint of your S3-compatible store).
  • The credential you will use can call s3:ListBucket on the bucket and s3:GetObject on the keys you want to expose. Scope the IAM policy to the prefix you are connecting.
  • For a non-AWS store (MinIO, Cloudflare R2, Wasabi), you have its endpoint URL and a matching access key pair.
  • For AWS Default Chain or a role assumed without static keys, the backend host must already carry credentials — an instance profile, IRSA, or the standard environment/config files.

​
Connect in Bag of words

Go to Data Sources → Add data source → Amazon S3 and fill in the form.
allowed_extensions and index_content are deprecated. Use include_globs with a pattern like **/*.pdf instead of allowed_extensions, and index_mode instead of index_content.

​
Indexing modes

Reads are always live; index_mode only decides what is cached ahead of time.

​
Authentication modes

All three modes are system-scope only — one shared credential serves every user of the connection. There is no per-user sign-in for S3; restrict who can use the connection with access rules instead. See Authentication and access.

​
AWS Access Key (aws_keys)

The default. Static IAM credentials.

​
AWS Assume Role (STS) (aws_role)

Bag of words calls sts:AssumeRole and uses the returned temporary credentials.

​
AWS Default Chain (aws_default)

No fields. Bag of words uses the standard AWS credential chain on the backend host — environment variables, shared config, instance profile, or IRSA. Use this when the deployment already has an AWS identity and you do not want to store keys in the product.

​
Troubleshooting

Check include_globs — patterns are matched relative to prefix, so a pattern that repeats the prefix will never match. Also confirm recursive is on if the objects sit below sub-prefixes.
The credential has s3:ListBucket but not s3:GetObject on those keys, or the object is outside include_globs — an off-glob key is denied, not just hidden.
Whole-object reads above max_file_mb are rejected. Ask the agent for a windowed read, or raise the limit.
The bucket exceeded max_catalog_objects. Narrow the connection with prefix or include_globs rather than raising the cap.