Visual overview
Glue discovers and transforms S3 data; Athena can query the resulting cataloged tables without loading them into a separate warehouse.
Technical reference
Glue separates metadata discovery from data transformation. Know which object owns which behavior.
Tables hold technical metadata such as schema, location, format, and partition information rather than copying the source dataset.
Crawlers scan supported stores; classifiers determine how data is recognized and converted into table metadata.
Glue can generate or run PySpark/Scala transformation logic and also supports other job/runtime options documented by AWS.
Bookmarks record processing state for supported sources/transforms so later job runs can identify data already processed.
These job properties control execution resources and failure behavior; they are independent of the Data Catalog schema.
Glue streaming ETL can process supported streaming sources continuously and write transformed results to supported destinations.
The Data Catalog stores metadata, not the source data
AWS Glue is a serverless data-integration service whose core pieces include the AWS Glue Data Catalog, crawlers, and managed ETL jobs. The Data Catalog is a persistent technical metadata store. Tables in the catalog describe where data is located and how it is structured; the underlying files or database rows remain in their original data stores. This separation lets multiple analytics engines share a common view of schemas without duplicating the dataset merely to register it.
Catalog databases group table definitions, while tables describe schema, formats, locations, and partition metadata. Services such as Athena, EMR, and Redshift Spectrum can use the catalog. That makes Glue important even in an architecture that never runs a Glue ETL job: a shared metastore can be the layer that makes S3 objects queryable as structured analytical tables.
Crawlers infer schema and update catalog tables
A Glue crawler connects to supported data stores, uses classifiers to recognize data formats and schema, and creates or updates tables in the Data Catalog. AWS supplies built-in classifiers, and custom classifiers can be defined where supported. Crawlers can run on demand or on schedules, which makes them useful when file layouts or partitions evolve over time.
Automatic discovery does not mean the inferred schema is always the business schema you want. Teams should design S3 prefixes, file formats, and schema evolution deliberately, and they can create catalog tables manually when precise control is required. For streaming sources, the Glue workflow differs from crawling persistent file stores: known schemas can be defined directly for streaming ETL rather than assuming that every source is crawler-driven.
- Catalog tables are metadata representations; cataloging does not copy the underlying dataset.
- Classifiers help crawlers infer file format and schema.
- Partitions recorded in metadata can help query engines avoid scanning unrelated S3 prefixes.
- Catalog permissions and data permissions remain separate security considerations.
ETL jobs execute transformation logic on managed infrastructure
Glue jobs run transformation logic using managed execution environments, including Apache Spark-based ETL. Jobs can read catalog tables or supported sources, transform records, and write to destinations such as S3 or analytical databases. Glue Studio provides a visual authoring environment, while scripts allow programmatic transformation logic when a pipeline needs finer control.
Operationally, job parameters such as worker type, worker count, timeout, retries, and job bookmarks affect behavior. Job bookmarks track previously processed input state for supported sources and transformations so recurring jobs can avoid reprocessing the same data unintentionally. Bookmarks are a processing-state feature, not a replacement for idempotent output design or data-quality checks.
Key takeaways
- 01
The Glue Data Catalog stores technical metadata while source data remains in its original location.
- 02
Crawlers inspect supported data sources and use classifiers to create or update catalog table definitions.
- 03
Athena, EMR, and Redshift Spectrum can use Glue Data Catalog metadata.
- 04
Glue ETL jobs provide managed transformation execution, commonly with Apache Spark.
- 05
Job bookmarks track previously processed input state for supported incremental ETL patterns.
Official AWS sources
Use these primary AWS resources for the source material behind this article and for deeper reference.