Faker
Sync overview
The Sample Data (Faker) source generates sample data using the python
mimesis package.
Output schema
This source will generate an "e-commerce-like" dataset with users, products, and purchases. Here's what is produced at a Postgres destination connected to this source:
CREATE TABLE "public"."users" (
"address" jsonb,
"occupation" text,
"gender" text,
"academic_degree" text,
"weight" int8,
"created_at" timestamptz,
"language" text,
"telephone" text,
"title" text,
"updated_at" timestamptz,
"nationality" text,
"blood_type" text,
"name" text,
"id" float8,
"age" int8,
"email" text,
"height" text,
-- "_airbyte_ab_id" varchar,
-- "_airbyte_emitted_at" timestamptz,
-- "_airbyte_normalized_at" timestamptz,
-- "_airbyte_users_hashid" text
);
CREATE TABLE "public"."users_address" (
"_airbyte_users_hashid" text,
"country_code" text,
"province" text,
"city" text,
"street_number" text,
"state" text,
"postal_code" text,
"street_name" text,
-- "_airbyte_ab_id" varchar,
-- "_airbyte_emitted_at" timestamptz,
-- "_airbyte_normalized_at" timestamptz,
-- "_airbyte_address_hashid" text
);
CREATE TABLE "public"."products" (
"id" float8,
"make" text,
"year" float8,
"model" text,
"price" float8,
"created_at" timestamptz,
-- "_airbyte_ab_id" varchar,
-- "_airbyte_emitted_at" timestamptz,
-- "_airbyte_normalized_at" timestamptz,
-- "_airbyte_dev_products_hashid" text,
);
CREATE TABLE "public"."purchases" (
"id" float8,
"user_id" float8,
"product_id" float8,
"purchased_at" timestamptz,
"added_to_cart_at" timestamptz,
"returned_at" timestamptz,
-- "_airbyte_ab_id" varchar,
-- "_airbyte_emitted_at" timestamptz,
-- "_airbyte_normalized_at" timestamptz,
-- "_airbyte_dev_purchases_hashid" text,
);
Features
| Feature | Supported?(Yes/No) | Notes |
|---|---|---|
| Full Refresh Sync | Yes | |
| Incremental Sync | Yes | |
| Namespaces | No |
Of note, if you choose Incremental Sync, state will be maintained between syncs, and once you hit
count records, no new records will be added.
You can choose a specific seed (integer) as an option for this connector which will guarantee that
the same fake records are generated each time. Otherwise, random data will be created on each
subsequent sync.
Requirements
None!
Reference
Config fields reference
Field
Type
Property name
integer
count
integer
seed
integer
records_per_slice
boolean
always_updated
integer
parallelism
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 6.1.3 | 2024-06-04 | 39029 | [autopull] Upgrade base image to v1.2.1 |
| 6.1.2 | 2024-06-03 | 38831 | Bump CDK to allow and prefer versions 1.x |
| 6.1.1 | 2024-05-20 | 38256 | Replace AirbyteLogger with logging.Logger |
| 6.1.0 | 2024-04-08 | 36898 | Update car prices and years |
| 6.0.3 | 2024-03-15 | 36167 | Make 'count' an optional config parameter. |
| 6.0.2 | 2024-02-12 | 35174 | Manage dependencies with Poetry. |
| 6.0.1 | 2024-02-12 | 35172 | Base image migration: remove Dockerfile and use the python-connector-base image |
| 6.0.0 | 2024-01-30 | 34644 | Declare 'id' columns as primary keys. |
| 5.0.2 | 2024-01-17 | 34344 | Ensure unique state messages |
| 5.0.1 | 2023-01-08 | 34033 | Add standard entrypoints for usage with AirbyteLib |
| 5.0.0 | 2023-08-08 | 29213 | Change all *id fields and products.year to be integer |
| 4.0.0 | 2023-07-19 | 28485 | Bump to test publication |
| 3.0.2 | 2023-07-07 | 27807 | Bump to test publication |
| 3.0.1 | 2023-06-28 | 27807 | Fix bug with purchase stream updated_at |
| 3.0.0 | 2023-06-23 | 27684 | Stream cursor is now updated_at & remove records_per_sync option |
| 2.1.0 | 2023-05-08 | 25903 | Add user.address (object) |
| 2.0.3 | 2023-02-20 | 23259 | bump to test publication |
| 2.0.2 | 2023-02-20 | 23259 | bump to test publication |
| 2.0.1 | 2023-01-30 | 22117 | source-faker goes beta |
| 2.0.0 | 2022-12-14 | 20492 and 20741 | Decouple stream states for better parallelism |
| 1.0.0 | 2022-11-28 | 19490 | Faker uses the CDK; rename streams to be lower-case (breaking), add determinism to random purchases, and rename |
| 0.2.1 | 2022-10-14 | 19197 | Emit AirbyteEstimateTraceMessage |
| 0.2.0 | 2022-10-14 | 18021 | Move to mimesis for speed! |
| 0.1.8 | 2022-10-12 | 17889 | Bump to test publish command (2) |
| 0.1.7 | 2022-10-11 | 17848 | Bump to test publish command |
| 0.1.6 | 2022-09-07 | 16418 | Log start of each stream |
| 0.1.5 | 2022-06-10 | 13695 | Emit timestamps in the proper ISO format |
| 0.1.4 | 2022-05-27 | 13298 | Test publication flow |
| 0.1.3 | 2022-05-27 | 13248 | Add options for records_per_sync and page_size |
| 0.1.2 | 2022-05-26 | 13248 | Test publication flow |
| 0.1.1 | 2022-05-26 | 13235 | Publish for AMD and ARM (M1 Macs) & remove User.birthdate |
| 0.1.0 | 2022-04-12 | 11738 | The Faker Source is created |