Batch Upserts and Performance Tuning

OL DataIQ Batch is designed for idempotent loads. It selects dialect-aware upsert behavior: ON CONFLICT DO NOTHING for PostgreSQL and SQLite, INSERT IGNORE for MySQL/MariaDB, and MERGE for Oracle, SQL Server, DB2, and H2.

Key columns are resolved from stored mappings, then target-table primary keys. If neither is available, the writer fails fast.

Duplicate-key and data-integrity exceptions can be skipped up to the configured limit (default: 100); skips appear in step metrics.

Default throughput settings

• Chunk size: 500
• CoreIQ page size: 500
• PostgreSQL batched-insert rewrite enabled where supported
• Small target Hikari pool: maximum 4, minimum idle 1

Override settings through the ODZ application.yml, command-line Spring properties, or environment variables such as BATCH_IMPORT_CHUNK_SIZE.