Migration on Discovery notebook

Inspecting a catalog with pg_migrate

Florent Jardin — Wed, 16 Sep 2026 09:00:00 +0200

Over the past few years, I have been working on assessing the strengths and weaknesses of PostgreSQL migration tools. Several articles published by yours truly led to the ambitious project I have been pursuing since 2024 with my colleagues Étienne Bersac and Pierre-Louis Gonon.

… And the fisrt stable version of PostgreSQL Migrator was released on September 4th. This is an opportunity to showcase features I use daily and what advantages they offer over other tools. In this article, I want to focus on one of them, particularly valuable when preparing a migration: the offline catalog.

Inspecting the Oracle catalog

The catalog of a relational database contains the structure of the data model, table column names and data types, constraint definitions, the definition of a view or a function, and so on. Everything declared by the user with DDL (Data Definition Language) is stored in the catalog as the single source of truth.

In systems like PostgreSQL, MySQL or MSSQL Server, the standard provides a universal catalog, the information_schema schema. For example, table names can be retrieved there with the same query:

SELECT table_name FROM information_schema.tables
 WHERE table_schema = 'scott'
 ORDER BY table_name;

However, this is not the ideal solution to reconstruct a data model. Each of these systems conforms to the SQL standard as best it can but very often enriches it with language extensions or takes liberties with the implementation of a feature. As a result, the information_schema catalog is not the universal source, little more than a set of views on top of each system’s proprietary system catalog.

Turning to Oracle and Ora2Pg. If we want to recreate the structure of a table in the Oracle ecosystem, several methods exist and they all rely on the catalog views which I cover last.

The DESCRIBE command

By far the least informative of the solutions but the fastest for a first inspection. It is analogous to the \d meta-command in psql or the pragma table_info in SQLite.

DESCRIBE SCOTT.EMP;

-- Name Null? Type
-- -------- -------- ------------
-- EMPNO NOT NULL NUMBER(4)
-- ENAME VARCHAR2(10)
-- JOB VARCHAR2(9)
-- MGR NUMBER(4)
-- HIREDATE DATE
-- SAL NUMBER(7,2)
-- COMM NUMBER(7,2)
-- DEPTNO NUMBER(2)

The DBMS_METADATA package

You can use a very handy Oracle package to extract the DDL of a table as SQL text. The equivalents in other systems are the SHOW CREATE TABLE statement in MySQL or .schema in SQLite.

SET pagesize 0
SET long 20000
SELECT dbms_metadata.get_ddl(
 object_type => 'TABLE',
 schema => 'SCOTT',
 name => 'EMP');

-- CREATE TABLE "SCOTT"."EMP"
-- ( "EMPNO" NUMBER(4,0),
-- "ENAME" VARCHAR2(10),
-- "JOB" VARCHAR2(9),
-- "MGR" NUMBER(4,0),
-- "HIREDATE" DATE,
-- "SAL" NUMBER(7,2),
-- "COMM" NUMBER(7,2),
-- "DEPTNO" NUMBER(2,0),
-- CONSTRAINT "PK_EMP" PRIMARY KEY ("EMPNO")
-- USING INDEX PCTFREE 10 INITRANS 2 MAXTRANS 255
-- STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
-- PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT)
-- TABLESPACE "SYSTEM" ENABLE,
-- CONSTRAINT "FK_DEPTNO" FOREIGN KEY ("DEPTNO")
-- REFERENCES "SCOTT"."DEPT" ("DEPTNO") ENABLE
-- ) PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING
-- STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
-- PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT)
-- TABLESPACE "SYSTEM"

Catalog views

The previous results are derived from several system views. Unsurprisingly, classic GUI tools like SQL Developer or DBeaver, as well as Ora2Pg and PostgreSQL Migrator, query these views directly to get the full DDL.

Ora2Pg fetches the table list from the ALL_OBJECTS view and stores it in memory in a hash $self->{tables}. Then, it iterates over ALL_TAB_COLUMNS to collect the details of each table and attach them as a nested level of that same global hash $self->{tables}{$table}{column_info}{$column}.

SELECT A.OWNER,A.OBJECT_NAME,A.OBJECT_TYPE FROM ALL_OBJECTS A
 WHERE A.OBJECT_TYPE IN ('TABLE','VIEW') AND A.OWNER='SCOTT';

SELECT A.COLUMN_NAME, A.DATA_TYPE, A.DATA_LENGTH, A.NULLABLE, A.DATA_DEFAULT,
 A.DATA_PRECISION, A.DATA_SCALE, A.CHAR_LENGTH, A.TABLE_NAME, A.OWNER
 FROM ALL_TAB_COLUMNS A
 WHERE A.TABLE_NAME NOT LIKE 'BIN$%' AND A.TABLE_NAME='EMP'
 ORDER BY A.COLUMN_ID

The in-memory $self structure maintained by Ora2Pg does not persist across runs. The source catalog must be re-read on every run. By contrast, I came across an alternative named db_migrator, based on Foreign Data Wrapper (FDW). With db_migrator, the catalog is stored in a PostgreSQL database.

It was then easy to snapshot the remote catalog with the db_migrate_refresh() function and to query it offline, without reconnecting to the Oracle instance. This discovery led to the first prototypes of the JSON catalog of PostgreSQL Migrator.

Querying the JSON catalog

Storing the catalog requires a structured, stable and flexible format. In practice, storing the data in PostgreSQL tables seemed elegant to DBA eyes but proved unsuitable. The PostgreSQL Migrator tool is designed to be portable and we did not want to pollute the final PostgreSQL instance with a temporary schema.

Like Ora2Pg, pg_migrate initializes a Catalog in-memory structure (internal/catalog/catalog.go) at the start of the inspection. This catalog can be extended to support Oracle or MySQL/MariaDB sources, to collect engine-specific objects such as packages or synonyms. Once the remote catalog has been read, it is then serialized to a JSON file. Currently, version 1.0 uses the sonic library.

Inspection is the first step to kick off a project with pg_migrate, the nickname for the command-line interface (CLI) of PostgreSQL Migrator. After initializing an empty directory and checking access to the remote Oracle database, the logs prompt us to run the pg_migrate inspect command.

$ ora-scott && cd ora-scott
$ pg_migrate init --source "oracle://system:manager@localhost:1521/freepdb1"
14:55:28 INFO Databases connections closed. source=1 target=0
14:55:28 INFO Initialized migration project. source="Oracle Database 23ai Free"
14:55:28 INFO Inspect source catalog using pg_migrate inspect.
$ pg_migrate --verbose inspect
14:56:03 DEBUG Cache miss. name=Source
14:56:03 INFO Inspecting source database. driver=oracle
14:56:03 INFO This may take some time.
14:56:04 DEBUG Inspected schemas. count=1 duration=84.735201ms
14:56:05 DEBUG Inspected roles. count=6 duration=465.814794ms
14:56:06 DEBUG Inspected sequences. count=0 duration=1.050025889s
14:56:07 DEBUG Inspected routines. count=0 duration=2.794483231s
14:56:07 DEBUG Inspected views. count=0 duration=2.63147026s
14:56:21 DEBUG Inspected tables. count=4 duration=15.720365706s
14:56:31 DEBUG Inspected columns. count=18 duration=10.735219841s
14:56:41 DEBUG Inspected keys. count=2 duration=19.945416304s
14:56:44 DEBUG Inspected tables-indexes. count=0 duration=23.287211484s
14:56:48 DEBUG Inspected foreign-keys. count=1 duration=6.525976802s
14:56:58 DEBUG Inspected checks. count=0 duration=10.622190194s
14:56:58 INFO Inspected schema. name=SCOTT
14:56:58 DEBUG Inspection completed. duration=54.968668546s
...
14:57:12 INFO Inspection terminated. errs=0 online=true
14:57:12 INFO Execute pg_migrate ui to browse project.
14:57:12 INFO Execute pg_migrate dump to migrate for PostgreSQL.

On first run, the local catalog does not exist (Cache miss. name=Source), which connects to the remote database to build it from scratch. Cache files are written to the hidden .pg_migrate directory. The source catalog is simply named Source.json.

$ tree .pg_migrate/
.pg_migrate
├── Info.json
├── Map.json
├── Source.json
├── Stats.json
└── Target.json

I then had to set aside my beloved SQL to learn the jq language in order to query the JSON files in a pg_migrate project. For example, the following filter lists all columns and their types for the SCOTT.EMP table:

$ set filter '.Tables[] | select (.Name == "EMP") | .Columns[] | .objectDefinition'
$ jq -r $filter < .pg_migrate/Source.json
EMPNO NUMBER(4, 0) NOT NULL
ENAME VARCHAR2(10)
JOB VARCHAR2(9)
MGR NUMBER(4, 0)
HIREDATE DATE
SAL NUMBER(7, 2)
COMM NUMBER(7, 2)
DEPTNO NUMBER(2, 0)

Of course, the tool has come a very long way since 2024 and this offline catalog architecture has become the backbone of the product. These files opened up use cases previously out of reach with Ora2Pg, in addition to the offline mode.

Conversion transforms a Source.json file into a Target.json file by applying generic and specific conversion rules;
An audit phase traverses every node to assign a score, plus annotations about conversion pitfalls;
A single-page (SPA) web interface lets you browse both models (source and target) without writing jq filters.

The tool ships with a dump command, which can also be used to display the DDL for an object stored in Target.json. Currently, the full command is as follows:

$ pg_migrate dump --force --section=pre-data Tables/scott.emp 2>/dev/null | cat
--
-- Name: emp; Type: TABLE; Schema: scott; Owner: scott
--

CREATE TABLE "scott"."emp" (
 "empno" smallint NOT NULL,
 "ename" varchar(10),
 "job" varchar(9),
 "mgr" smallint,
 "hiredate" timestamp,
 "sal" numeric(7, 2),
 "comm" numeric(7, 2),
 "deptno" smallint
);

ALTER TABLE "scott"."emp" OWNER TO "scott";

Conclusion

I am deeply proud of the work accomplished with this first stable release. With my colleagues at Dalibo, we are wrapping up a thrilling adventure, more than two years spent developing a free, libre and open-source tool meant to surpass its illustrious predecessor Ora2Pg. The roadmap is clear. Starting this fall, we begin the next phase: procedural code conversion. The foundations are solid, the Go language lets us address real needs for simplification and modernity.

At a time when digital empires harvest our operational data, capturing the value of our companies’ work, it is urgent to react and to believe that another digital world is possible. Migration projects are becoming complex and demanding, involving very diverse roles and skills, and we aim to provide an alternative that does not rely exclusively on LLM agents or American services.

Free software has shown the way for decades; what we lack is the bridges to move away from these technologies that deprive us of freedom, innovation and means. At my level as a DBA and Product Owner, committed to the Dalibo cooperative, I contribute to this collective effort.

Substituting a variable in a SQL script

Florent Jardin — Mon, 25 Nov 2024 09:00:00 +0100

In a world where we constantly seek to automate repetitive tasks, it is common to write down a query in a script, make it more convenient, and eventually integrate the whole thing into a project’s codebase. Tools like SQL*Plus and psql can be powerful allies in this game, as relevant as Bash or Python interpreters.

In several projects I have been involved in, I have come across a large number of those kinds of scripts. Some of them have the particularity of offering input parameters, processed by SQL*Plus with the very comfortable mechanism named variable substitution. In this article, I share some tips to convert them to an equivalent syntax that PostgreSQL’s psql tool can parse and manage.

Substituting a variable…

As a starting point, let’s look at the syntax supported by Oracle’s tool with a progressive example. We stand in the shoes of a user who wants to get the result of sales from a dummy company, applying two filters based on a product identifier and two dates passed as parameters.

-- sqlplus-report-01.sql
CONNECT user/password@database

DEF product_id = '&1'
DEF start_date = '&2'
DEF end_date = '&3'

SELECT NVL(SUM(amount), 0) AS total_amount
 FROM orders
 JOIN products USING (product_id)
 WHERE product_id = &product_id
 AND order_date BETWEEN TO_DATE('&start_date', 'YYYY-MM-DD')
 AND TO_DATE('&end_date', 'YYYY-MM-DD');

QUIT

$ sqlplus -S /nolog @sqlplus-report-01.sql 20 2024-11-01 2024-11-30
old 3: WHERE product_id = &product_id
new 3: WHERE product_id = 20
old 4: AND order_date BETWEEN TO_DATE('&start_date', 'YYYY-MM-DD')
new 4: AND order_date BETWEEN TO_DATE('2024-11-01', 'YYYY-MM-DD')
old 5: AND TO_DATE('&end_date', 'YYYY-MM-DD')
new 5: AND TO_DATE('2024-11-30', 'YYYY-MM-DD')

TOTAL_AMOUNT
------------
 1378.98

As shown above, the substitution takes place in SQL*Plus as soon as the & symbol is encountered, whether its content is surrounded by single quotes or not. At the beginning of the script, I applied a simple strategy by defining each variable with a meaningful name, to facilitate the readability and maintenance of the script. We don’t want to rely on variables named according to their position in the list of arguments.

psql can provide a similar feature, but the syntax is slightly different. We have to assign variables outside the script, using the -v or --set option. These variables will then be substituted in the script using the interpolation system.

-- psql-report-01.sql
\pset footer off

SELECT COALESCE(SUM(amount), 0) AS total_amount
 FROM orders
 JOIN products USING (product_id)
 WHERE product_id = :product_id
 AND order_date BETWEEN :'start_date'::date
 AND :'end_date'::date;

$ psql -f psql-report-01.sql \
$ -v product_id=20 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
 total_amount
--------------
 1378.98

From this point, we can see that the conversion is quite straightforward. The substitution syntax is similar with the : character instead of &. However, if the variable states as a literal string, we must use the : character outside the quotes and not inside.

… in an anonymous block

Things get complicated when the script becomes more complex and requires us to manipulate our previous variables in an anonymous PL/SQL block. Let’s take our script back to enhance it with a personalized message if the requested product does not exist. A first read on the products table should eventually raise the NO_DATA_FOUND exception that we want to catch.

-- sqlplus-report-02.sql
CONNECT user/password@database

DEF product_id = '&1'
DEF start_date = '&2'
DEF end_date = '&3'

SET serveroutput ON
SET feedback OFF
SET verify OFF

DECLARE
 p_id products.product_id%TYPE;
 p_sum NUMBER;
 p_start DATE := TO_DATE('&start_date', 'YYYY-MM-DD');
 p_end DATE := TO_DATE('&end_date', 'YYYY-MM-DD');

BEGIN
 SELECT product_id INTO p_id
 FROM products
 WHERE product_id = &product_id;

 SELECT NVL(SUM(amount), 0) INTO p_sum
 FROM orders
 WHERE product_id = p_id
 AND order_date BETWEEN p_start AND p_end;

 DBMS_OUTPUT.PUT_LINE('Total amount: ' || p_sum);

EXCEPTION
 WHEN NO_DATA_FOUND THEN
 DBMS_OUTPUT.PUT_LINE('Product ' || &product_id || ' does not exist');
END;
/

QUIT

$ sqlplus -S /nolog @sqlplus-report-02.sql 120 2024-11-01 2024-11-30
Product 120 does not exist

Unfortunately, our beloved psql command-line interface does not support the ubstitution of variables in an anonymous block and any attempt to do so will result in a syntax error.

-- psql-report-02-wrong.sql
DO $$
DECLARE
 p_id products.product_id%TYPE;
 p_sum numeric;
 p_start date := :'start_date'::date;
 p_end date := :'end_date'::date;

BEGIN
 SELECT product_id INTO STRICT p_id
 FROM products
 WHERE product_id = :product_id;

 SELECT COALESCE(SUM(amount), 0) INTO p_sum
 FROM orders
 WHERE product_id = p_id
 AND order_date BETWEEN p_start AND p_end;

 RAISE NOTICE 'Total amount: %', p_sum;

EXCEPTION
 WHEN no_data_found THEN
 RAISE NOTICE 'Product % does not exist', :product_id;
END;
$$;

$ psql -f psql-report-02-wrong.sql \
$ -v product_id=120 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
psql:psql-report-02-wrong.sql:25: ERROR: syntax error at or near ":"
LINE 5: p_start date := :'start_date'::date;
 ^

Do not panic! psql provides us with a way to achieve the same result, with a deeper rewrite of the script. We will use the \gset meta-command to store the product state and then the \if meta-command to perform the control.

-- psql-report-02.sql
\pset footer off

SELECT NOT EXISTS(
 SELECT product_id
 FROM products
 WHERE product_id = :product_id
) AS unknown_product \gset

\if :unknown_product
 \echo 'Product' :product_id 'does not exist'
 \quit
\endif

SELECT COALESCE(SUM(amount), 0) AS total_amount
 FROM orders
 JOIN products USING (product_id)
 WHERE product_id = :product_id
 AND order_date BETWEEN :'start_date'::date
 AND :'end_date'::date;

$ psql -f report.sql \
$ -v product_id=120 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
Product 120 does not exist

… at any costs

In the wild, it may happen that psql meta-commands are not enough to handle all the intricacies (and absurdities) that SQL*Plus-compatible scripts can present. Let’s move on with a more complex need we want to address.

Our user now wants to get a performance score for the sales period of his product, comparing it with the previous period. We would need a cursor here to reuse the same sales calculation query multiple times and a function to compute a score by handling the possible division by zero.

-- sqlplus-report-03.sql
CONNECT user/password@database

DEF product_id = '&1'
DEF start_date = '&2'
DEF end_date = '&3'

SET serveroutput ON
SET feedback OFF
SET verify OFF

DECLARE
 p_id products.product_id%TYPE;
 p_sum NUMBER;
 p_prev_sum NUMBER;
 p_start DATE := TO_DATE('&start_date', 'YYYY-MM-DD');
 p_end DATE := TO_DATE('&end_date', 'YYYY-MM-DD');
 p_prev_start DATE := p_start - (p_end - p_start);
 p_prev_end DATE := p_start - 1;

 CURSOR c_orders (v_start DATE, v_end DATE) IS
 SELECT SUM(amount) AS total_amount
 FROM orders
 WHERE product_id = p_id
 AND order_date BETWEEN v_start AND v_end;

 FUNCTION score(p_sum NUMBER, p_prev_sum NUMBER)
 RETURN NUMBER IS
 v_score NUMBER;
 BEGIN
 RETURN ROUND((p_sum - p_prev_sum) / p_prev_sum * 100, 2);
 EXCEPTION
 WHEN ZERO_DIVIDE THEN
 RETURN NULL;
 END;

BEGIN
 SELECT product_id INTO p_id
 FROM products
 WHERE product_id = &product_id;

 OPEN c_orders(p_start, p_end);
 FETCH c_orders INTO p_sum;
 CLOSE c_orders;

 OPEN c_orders(p_prev_start, p_prev_end);
 FETCH c_orders INTO p_prev_sum;
 CLOSE c_orders;

 DBMS_OUTPUT.PUT_LINE('Total amount: ' || p_sum);
 DBMS_OUTPUT.PUT_LINE('Performance score: ' || score(p_sum, p_prev_sum));

EXCEPTION
 WHEN NO_DATA_FOUND THEN
 DBMS_OUTPUT.PUT_LINE('Product ' || &product_id || ' does not exist');
END;
/

QUIT

$ sqlplus -S /nolog @sqlplus-report-03.sql 20 2024-11-01 2024-11-30
Total amount: 1378.98
Performance score: 162.99

Of course, the example is deliberately simplistic and could be addressed by a window function, but we will play the game of converting the script as close as possible to its original syntax. To do this, let’s identify the weaknesses of PL/pgSQL.

Temporary routines are not supported

PL/pgSQL language does not allow the declaration of stored functions or storeed procedures inside an anonymous block. The only alternative is to define it globally, so that it is known to the parser and executed correctly. In this present case, I do not want the function to persist after the call of my script. No problem, we can create a temporary function in the pg_temp schema!

-- temp-function.sql
DO $$
BEGIN
 CREATE FUNCTION pg_temp.score(p_sum numeric, p_prev_sum numeric)
 RETURNS numeric LANGUAGE plpgsql AS $func$
 BEGIN
 RETURN ROUND((p_sum - p_prev_sum) / p_prev_sum * 100, 2);
 EXCEPTION
 WHEN division_by_zero THEN
 RETURN NULL;
 END;
 $func$;

 RAISE NOTICE 'Score: %', pg_temp.score(100.0, 50.0);
 RAISE NOTICE 'Score: %', pg_temp.score(100.0, 0);
END;
$$;

$ psql -f temp-function.sql
psql:temp-function.sql:16: NOTICE: Score: 100.00
psql:temp-function.sql:16: NOTICE: Score: <NULL>
DO

The pg_temp.score(numeric, numeric) function is now accessible within the block, and only for the duration of the session. Exception handling respects the expressed need. We just had to replace ZERO_DIVIDE with division_by_zero as describe in the documentation.

SQL substitution is not supported

Our main issue strikes back: substitution does not occur when we are in a PL/pgSQL block. To work around this limitation, we will have to use the custom session variables made available for the PostgreSQL extension ecosystem.

Those variables can be manipulated as well in psql with the SET keyword as in SQL or PL/pgSQL with the current_setting and set_config functions. The only constraint is to define a prefix that does not conflict with that of other variables already declared in our instance.

It means that our parameters can be set as session variables in the very beginning of the script. As soon as we are in an anonymous block, we must then assign them back to regular variables as shown in the following code:

SET my.product_id = :product_id;
SET my.start_date = :start_date;
SET my.end_date = :end_date;

DO $$
DECLARE
 my_id int := current_setting('my.product_id')::int;
 p_start date := current_setting('my.start_date')::date;
 p_end date := current_setting('my.end_date')::date;
BEGIN
 ...
END;
$$;

Full conversion

A new version of the script, converted from PL/SQL to PL/pgSQL, can be obtained with all previous tricks put together:

-- psql-report-03.sql
\set QUIET on

SET my.product_id = :product_id;
SET my.start_date = :'start_date';
SET my.end_date = :'end_date';

DO $$
DECLARE
 my_id int := current_setting('my.product_id')::int;
 p_id products.product_id%TYPE;
 p_sum numeric;
 p_prev_sum numeric;
 p_start date := current_setting('my.start_date')::date;
 p_end date := current_setting('my.end_date')::date;
 p_prev_start date := p_start - interval '1 day' * (p_end - p_start);
 p_prev_end date := p_start - interval '1 day';

 c_orders CURSOR (v_start date, v_end date) IS
 SELECT COALESCE(SUM(amount), 0)
 FROM orders
 WHERE product_id = p_id
 AND order_date BETWEEN v_start AND v_end;

BEGIN
 CREATE FUNCTION pg_temp.score(p_sum numeric, p_prev_sum numeric)
 RETURNS numeric LANGUAGE plpgsql AS $func$
 BEGIN
 RETURN ROUND((p_sum - p_prev_sum) / p_prev_sum * 100, 2);
 EXCEPTION
 WHEN division_by_zero THEN
 RETURN NULL;
 END;
 $func$;

 SELECT product_id INTO STRICT p_id
 FROM products
 WHERE product_id = my_id;

 OPEN c_orders(p_start, p_end);
 FETCH c_orders INTO p_sum;
 CLOSE c_orders;

 OPEN c_orders(p_prev_start, p_prev_end);
 FETCH c_orders INTO p_prev_sum;
 CLOSE c_orders;

 RAISE NOTICE 'Total amount: %', p_sum;
 RAISE NOTICE 'Performance score: %', pg_temp.score(p_sum, p_prev_sum);

EXCEPTION
 WHEN no_data_found THEN
 RAISE NOTICE 'Product % does not exist', my_id;
END;
$$;

$ psql -f psql-report-03.sql\
$ -v product_id=20 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
psql:psql-report-03.sql:55: NOTICE: Total amount: 1378.98
psql:psql-report-03.sql:55: NOTICE: Performance score: 162.99

Conclusion

This kind of rewriting becomes affordable when we know the right tricks, without losing sight of the initial goal. Of course, a one-to-one translation, by preserving the syntax and the original intent, could be more challenging than that blog post example.

I’m a bit skeptical about those who want to keep such development practices. They seem outdated to me. Isn’t migrating to PostgreSQL an opportunity to modernize our codebase or to question our relationship with technology? In our example, as I previously mentioned, it is possible to get the same result in a single SQL, combining meta-commands, variables substitution, common-table expressions, and a the LAG window function.

In that way, we throw away the PL/pgSQL block and its limitations.

-- psql-report-04.sql
\pset footer off

SELECT NOT EXISTS(
 SELECT product_id
 FROM products
 WHERE product_id = :product_id
) AS unknown_product \gset

\if :unknown_product
 \echo 'Product' :product_id 'does not exist'
 \quit
\endif

SELECT :'end_date'::date - :'start_date'::date AS days \gset
SELECT :'start_date'::date - :days AS prev_start \gset

WITH periods AS (
 SELECT d AS start_date, d + :days * '1 day'::interval AS end_date
 FROM generate_series(:'prev_start'::date,
 :'start_date'::date,
 :days * '1 day'::interval) AS s(d)
), amounts AS (
 SELECT start_date, COALESCE(SUM(amount), 0) AS total_amount,
 LAG(SUM(amount), 1) OVER (ORDER BY start_date) AS prev_total_amount
 FROM orders
 JOIN periods ON order_date BETWEEN start_date AND end_date
 WHERE product_id = :product_id
 GROUP BY start_date
)
SELECT total_amount,
 ROUND((total_amount - prev_total_amount) /
 prev_total_amount * 100, 2) AS performance_score
 FROM amounts
 WHERE prev_total_amount IS NOT NULL;

$ psql -f psql-report-04.sql\
$ -v product_id=20 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
 total_amount | performance_score
--------------+-------------------
 1378.98 | 162.99

You can find all the scripts of this article, as well as the DDL commands to the data model, at this address.

An assistant to copy data from a remote server

Florent Jardin — Tue, 28 May 2024 00:00:00 +0000

During the last PGSession organized by Dalibo, I wrote and led a workshop (french) on the migration to PostgreSQL using Foreign Data Wrappers, or FDW. This was an opportunity to present to the general public the db_migrator extension for which I wrote an article on this blog.

While working on this workshop, we noticed that copying data with the db_migrator extension is not perfectly supported. Indeed, although there is a low-level function to distribute the transfer table by table over several processes, many situations will require writing a large number of SQL queries to get out of trouble. Over the following months, I worked on the design of an assistant written in PL/pgSQL whose purpose is to simplify the generation of these queries.

Data transfer without an assistant

There is no magic in the operation of copying data through a Foreign Data Wrapper. Everything boils down to a series of INSERT queries that must be executed in a predefined order. Let’s take the 16 tables of the well-known “Sakila” data model (available at this address) to illustrate the need for generating transfer queries.

For the purpose of this article, I have a PostgreSQL database with the two following schemas that I will use in my demonstrations:

mysql: the source schema containing the definition of the external tables via the mysql_fdw extension;
public: the target schema where the data will be copied.

Each external table is subject to a quick study to obtain the correct column type match, and their definition is kept in an SQL file for the project teams. For example, the rental foreign tables is defined as follows:

CREATE FOREIGN TABLE mysql.rental (
 rental_id integer NOT NULL,
 rental_date timestamp without time zone NOT NULL,
 inventory_id integer NOT NULL,
 customer_id smallint NOT NULL,
 return_date timestamp without time zone,
 staff_id smallint NOT NULL,
 last_update timestamp without time zone NOT NULL
)
SERVER sakila_mysql
OPTIONS (
 dbname 'sakila',
 table_name 'rental'
);

When creating the public.rental table that will host the data, it is opportune to decide whether we want to set up partitioning, something that db_migrator is able to identify and set up. For the example, I take back the strict structure using the CREATE TABLE LIKE syntax to create all my target tables.

CREATE TABLE public.actor (LIKE mysql.actor);
CREATE TABLE public.address (LIKE mysql.address);
...

Even before setting up an INSERT query generator, it is easy to see the look of these queries. Each line of the external table will be read through a global SELECT, then inserted into the target table. The migration script contains 16 instructions, one for each table.

-- insert.sql
INSERT INTO public.actor SELECT * FROM mysql.actor;
INSERT INTO public.address SELECT * FROM mysql.address;
...
INSERT INTO public.store SELECT * FROM mysql.store;

To benefit from several processes, I appreciate the xargs tool which allows distributing each line of the insert.sql file on a new psql session. This technique was presented in the February workshop, especially to parallelize the construction of indexes and primary keys, defined in an SQL file.

$ xargs -P 4 -a insert.sql -d '\n' -I % sh -c 'psql -c "%"'
INSERT 0 16
INSERT 0 603
...
INSERT 0 16044

This method is not perfect, it lacks verbosity, the SQL queries are static. Anyway, let’s see the next part to discover what my assistant can bring.

Demonstration of the assistant

Unlike my other PL/pgSQL projects, this assistant is not an extension and must be installed as a simple script. Once downloaded, it is sufficient to invoke it on the database of your choice with the following command:

$ psql -d sakila -f fdw-assistant.sql

The default schema is named assistant and contains a configuration table simply called config. For each table to be migrated, one must insert a single line that will serve as the starting point for generating data migration queries. In the current version, we find the following parameters:

source: the external table containing the data to be copied;
target: the target table where the data will be copied;
pkey: the primary key column of the target table;
priority: the lowest values define the tables to be processed first;
parts: the number of processes to launch for copying the data;
trunc: an option to empty the target table before copying the data;
condition: a WHERE clause to filter the data to be copied;
batchsize: the number of rows to copy before performing an intermediate COMMIT.

To initialize this table in the first instance, it is necessary to know at least the primary key columns of each remote table. By reporting the information from the relational diagram of the Sakila database, we can fill the config table as follows:

INSERT INTO assistant.config (source, target, pkey)
VALUES
 ('mysql.actor', 'public.actor', 'actor_id'),
 ('mysql.address', 'public.address', 'address_id'),
 ...
 ('mysql.store', 'public.store', 'store_id');

For each transfer, we indicate the source and target table, as well as the primary key column. The latter is required to sort the rows, split the transfer into several batches, and restart the transfer in case of interruption.

With this configuration, we can move on to the planning step. The stage and job tables are fed with new elements that will be used to drive and track the different transfers to be done.

SELECT * FROM assistant.plan();

 target | invocation
---------------+--------------------------
 customer | CALL assistant.copy(1);
 address | CALL assistant.copy(2);
 ...
(16 rows)

A view named report allows you to follow the progress of the different steps by joining the stage and job tables. It provides very useful elements to monitor the progress and throughput of the transfers.

SELECT target, state, rows FROM assistant.report WHERE stage_id = 1;

 target | state | rows
---------------+---------+------
 rental | pending | 0
 actor | pending | 0
 ...
(16 rows)

Rows returned by the plan() command can then be invoked one after the other with the \gexec meta-command of psql, or by using the file technique and distributing the queries with xargs.

Calling the copy() method is responsible for building the INSERT statement for copying data from a remote table to a local table. For example, for the customer table, the result of the call will be as follows:

CALL assistant.copy(1);

NOTICE: Executing: TRUNCATE public.customer
NOTICE: Executing: SELECT count(customer_id) FROM mysql.customer
 WHERE customer_id > 0
NOTICE: Executing: INSERT INTO public.customer
 SELECT customer_id, store_id, first_name, last_name, email,
 address_id, active, create_date, last_update
 FROM mysql.customer
 WHERE customer_id > 0 ORDER BY customer_id
CALL

At the end of the transfer, the report view provides a summary of the operation.

SELECT * FROM assistant.report
 WHERE stage_id = 1 AND target = 'public.customer'::regclass;

-[ RECORD 1 ]-------------------------
stage_id | 1
target | public.customer
job_start | 2024-05-28 10:19:18.334917
state | completed
rows | 599
total | 599
elapsed | 00:00:00.081273
rate | 7370.22
progress | 1.00
eti | 00:00:00
eta | 2024-05-28 10:19:18.334917

Distribution over several processes

As the assistant was designed, I felt the need to enrich the underlying queries to respond to other recurring use cases in the data migration domain. Among these, we find the ability to distribute the rows of the same table over several sessions, each with a WHERE clause based on the result of the Euclidean division (modulo) of the primary key.

To activate this feature, simply fill in the parts parameter in the config table. For example, for the film table, we can define:

UPDATE assistant.config SET parts = 4 WHERE target = 'public.film'::regclass;

This configuration takes effect during the next planning. The call to the plan() method then inserts four lines into the job table for the film table, each attached to a value between 0 and 3. The condition column is then enriched for the rows of the task table in view of the next step.

SELECT invocation FROM assistant.plan('{public.film}');

 invocation
--------------------------
 CALL assistant.copy(17);
 CALL assistant.copy(18);
 CALL assistant.copy(19);
 CALL assistant.copy(20);
(4 rows)

SELECT job_id, job.target, state, part, condition
 FROM assistant.job JOIN assistant.task USING (job_id)
 WHERE stage_id = 2;

 job_id | target | state | part | condition
--------+--------+---------+------+-----------------
 17 | film | pending | 0 | film_id % 4 = 0
 18 | film | pending | 1 | film_id % 4 = 1
 19 | film | pending | 2 | film_id % 4 = 2
 20 | film | pending | 3 | film_id % 4 = 3

When calling the copy() method, the assistant builds the INSERT queries based on the conditions previously defined. For example, for the first part of the film table, the trace indicates the INSERT queries that were generated.

CALL assistant.copy(17);

NOTICE: Executing: TRUNCATE public.film
NOTICE: Executing: SELECT count(film_id) FROM mysql.film
 WHERE film_id > 0 AND film_id % 4 = 0
NOTICE: Executing: INSERT INTO public.film
 SELECT film_id, title, description, release_year,
 language_id, original_language_id, rental_duration,
 rental_rate, length, replacement_cost, rating,
 special_features, last_update
 FROM mysql.film WHERE film_id > 0 AND film_id % 4 = 0
 ORDER BY film_id
CALL

The TRUNCATE operation only occurs for the session whose value of part is equal to 0. In the nominal case, this session is launched before all the others to respect the expected behavior in the configuration with the trunc column (true by default).

The obvious interest of this method is to obtain the best insertion rate for a given table, relying on the extraction power of the remote server and the writing capacity of the local server. The rate can be consulted with the report view, especially to compare two loads for the same table, like the example of the film table. The rate column is expressed in number of rows per second.

SELECT stage_id, target, state, rate FROM assistant.report
 WHERE target = 'public.film'::regclass;

 stage_id | target | state | rate
----------+-------------+-----------+----------
 1 | public.film | completed | 19162.59
 2 | public.film | completed | 51389.37

Restart after interruption

The ability to restart a copy in batch mode in case of interruption is one of the reasons why the primary key must be filled in the configuration. The assistant relies on the last extracted primary key value (using a RETURNING clause) to know the next restart point. In designing this feature, I had to arbitrate the limitations imposed by this mechanism.

Composite primary keys are not allowed, as the RETURNING clause can only return a single value;
Tables with composite primary keys cannot benefit from batch processing, and therefore cannot be restarted after interruption;
Data is systematically sorted during extraction, even if batch processing is not enabled;
The primary key column must be of numeric type.

The activation of a batch processing consists of updating the batchsize column of the configuration table. Let’s take the example of the rental table:

UPDATE assistant.config SET batchsize = 1000
 WHERE target = 'public.rental'::regclass;

SELECT invocation FROM assistant.plan('{public.rental}');

 invocation
--------------------------
 CALL assistant.copy(21);
(1 row)

The data transfer from the rental table is then divided into batches of 1000 rows. It is of course possible to combine this technique with parallelization, the WHERE clause will do most of the work of repartition to prevent the same line from being exported twice.

CALL assistant.copy(21);

TRUNCATE public.rental
SELECT count(rental_id) FROM mysql.rental WHERE rental_id > 0
INSERT INTO public.rental SELECT rental_id, ...
 FROM mysql.rental WHERE rental_id > 0 ORDER BY rental_id LIMIT 1000
...
...
INSERT INTO public.rental SELECT rental_id, ...
 FROM mysql.rental WHERE rental_id > 16005 ORDER BY rental_id LIMIT 1000
INSERT INTO public.rental SELECT rental_id, ...
 FROM mysql.rental WHERE rental_id > 16049 ORDER BY rental_id LIMIT 1000
CALL

As soon as a query no longer returns any rows, the assistant considers that the transfer is complete. The job table is updated at each iteration to follow the last value of the primary key sequence.

SELECT target, lastseq FROM assistant.job
 WHERE stage_id = 3 AND target = 'public.rental'::regclass;

 target | lastseq
---------------+---------
 public.rental | 16049

In case of interruption, it is possible to restart the transfer by calling the copy() method with the same task identifier. The assistant takes care of resuming the transfer from the last known primary key value.

CALL assistant.copy(21);

NOTICE: Executing: SELECT count(rental_id) FROM mysql.rental
 WHERE rental_id > 16049
NOTICE: Executing: INSERT INTO public.rental SELECT rental_id, ...
 FROM mysql.rental WHERE rental_id > 16049
 ORDER BY rental_id LIMIT 1000
CALL

Conclusion

The design of such a tool was a small personal challenge in line with my research on migration to PostgreSQL with the exclusive help of Foreign Data Wrappers. My main source of inspiration remains the Ora2Pg project, one of the most advanced open-source tools to date in the field of migration.

I am aware of the technical limitations of this assistant, and the tinkering that remains to be done to make life easier for a consultant like me. In another article, I would like to present another tool called dispatch that I have been maintaining for some time and with which I answer questions about orchestration and traceability of migration steps.

By taking a step back, the basic concepts are there, any other tool in other languages could perfectly emerge and enrich the open-source ecosystem in the quest for migration to PostgreSQL.

Transfer modes in a migration

Florent Jardin — Sat, 11 Nov 2023 00:00:00 +0000

In computer science, a migration project involves changing one or more technical components without impacting any application behaviors. In the realm of databases (and the profession I practice), it will involve choosing a new system (such as PostgreSQL) to replace another (such as Oracle or Microsoft SQL Server).

In a previous article (French), I described exhaustive steps to perform a complete migration using Foreign Data Wrappers technology. However, the critical data transfer step described there does not suit all situations. Let’s explore together the alternatives that cover a large portion of the needs.

The illustrations in this article depict a simple migration project scenario where data is transferred from one system to another between two data centers (dc1 to dc2). In a situation where the data does not leave the hosting infrastructure, it is a straightforward transfer between two separate servers.

Transfer without a network

Several reasons can justify that the two servers are not in direct connection with each other. This includes the prohibition of opening access to the source instance from the internet or the complexity of configuring a secure link (VPN) between the two infrastructures.

In such a scenario, it becomes necessary to dump the data outside the database while ensuring the consistency of the data. A physical backup of the database or an export in SQL format are reliable means to consolidate a complete archive of the database.

The transfer step is carried out according to the imagination of the teams. The simplest method is to transfer to an SFTP repository provided by the second data center or any other protocol that checks the integrity of the archive once transferred. The most unlikely method I have observed was the physical movement of a copy of the archive by a specialized courier company, by car, scooter, or plane depending on the distance to cover.

Once the archive is received by the teams and deemed intact, it may be necessary to import the data into a buffer database, provided that the file format (SQL or proprietary) is not yet compatible with PostgreSQL. From this precise moment, it is then possible to import the data into the target database respecting the order of object creations and data insertion.

Operation Summary:

Very high downtime (from several hours to several days);
Implementation complexity is low with the right tools;
Risk of data consistency issues is very low if integrity checks are performed at each step.

Transfer with intermediary

When the systems are hosted in the same location, setting up routing rules allows for easier connection between the two servers. Among the migration tools, we find the family of ETL (Extract Transform Load) tools that act as intermediaries between the two databases.

These tools provide a large number of connection drivers as well as data transformation methods to orchestrate migration without the need to export any table to a flat file. The bulk of the work is done in memory and with the help of multiple processors to speed up transfer operations.

Although specialized for migrations to PostgreSQL, I classify the open-source tool Ora2Pg in the category of ETL. It precisely meets the previous definition by opening multiple connections to the source instance to read tables in batches of 10,000 rows (using the DATA_LIMIT directive) and opening other connections to the target instance to insert them with COPY instructions, thanks to the pg_putcopydata method of the DBD::Pg driver.

Operation Summary:

High downtime (several hours);
Low implementation complexity with the right tools;
Very low risk to data consistency.

Direct transfer

This mode should feel familiar if you are a regular reader (subscribe!) because direct transfer between a third-party system and PostgreSQL relies on the technology of Foreign Data Wrappers.

This solution opens a direct channel between the PostgreSQL instance and the remote instance through external tables. Thus, it becomes possible to query the data with SQL, provided that a wrapper has been developed to communicate with the correct driver.

The extraction, transfer, and insertion operations are performed in the same transaction: it’s a simple INSERT INTO SELECT query. This method is faster than the previous mode since it eliminates a resource-intensive intermediary (ETL).

Although appealing, copying through a wrapper can be particularly slow for large data (such as Oracle’s BLOBs) because they have a different structure. It is then necessary to mix solutions: one with a direct transfer for tables without large data and the other with an ETL to optimize the volume of rows and the amount of memory to allocate.

Operation Summary:

Downtime: High (from several minutes to several hours);
Implementation Complexity: Low to High for large data;
Risk to Data Consistency: Very Low.

Partial catch-up transfer

For very large volumes where service interruption must be kept to a minimum, the solutions mentioned above can be very limiting. Exporting/importing the entire dataset is a lengthy and unavoidable step.

What I refer to as the “partial catch-up transfer” involves identifying the largest tables with mostly dead rows. It requires assurance that no UPDATE or DELETE modifications occur from the development team and that the table has a primary key where new values during an insertion are always greater than the previous.

Thus, an initial copy of the data can be performed live, without service interruption, where only the dead rows will be moved to the new system. Depending on the proportion of the complete volume, this step can save dozens of hours in the final operation. In addition to the transfer operations, it is necessary to maintain a register of primary key values that separates dead data from live (or upcoming) data. For each table, this value will be a kind of checkpoint for the catch-up phase.

During the crucial switch step, all live tables will be entirely copied, and the rows from dead tables whose key is greater than the memorized value will be selected and then inserted into their respective table. A preliminary study must determine whether indexes deserve to be created during the first or second transfer step, depending on the effective time they save or consume throughout the entire operation.

Operation Summary:

Low downtime (several tens of minutes);
High implementation complexity;
High risk of data consistency issues if the development team does not provide sufficient guarantees in choosing dead tables.

Transfer with transaction replay

Speaking of “transaction replay” is essentially referring to physical replication or logical replication. Each system has its own mechanisms, and the tools to implement them can vary. I recommend looking into excellent projects like ora_migrator, which implements replication using triggers with Oracle, and pg_chameleon, which decodes transaction logs from MySQL.

The main advantage of this transfer mode lies in the system’s ability to log all modifications requested and record them in a sequence of transactions. Thus, between time T0 and time T1, it becomes possible to reproduce all changes to achieve a consistent and faithful state.

In this scenario, the operation requires a full and coherent data load into the target database. This initialization phase occurs without service interruption but requires particular care in monitoring disks or system source alerts because capturing data maintains a snapshot that can become costly if the transfer takes too long.

At the end of initialization, the replay can start without penalizing the production source database’s activity. The tool consumes each event in the order of arrival to transform it into an insert (INSERT), update (UPDATE), or delete (DELETE) in the PostgreSQL database. The cutover operation then consists of prohibiting any new modification to the source system, waiting for the end of replay, and changing the connection strings so that applications connect to the new PostgreSQL database.

Operation Summary:

Very low downtime (a few seconds);
Implementation complexity is high, even very high depending on the chosen solutions;
Risk of data consistency issues is very high if the tool fails for a transaction it cannot decode during the replay step.

Freedom of Choice

This comprehensive review had been on my mind for some time, and it seemed necessary to shed light on the various constraints posed by a migration project. In the sphere of consultants I interact with and the open-source projects I actively monitor, I try to convince myself that there could be a universal solution capable of addressing every need and situation outlined in this article.

I deliberately avoided discussing the Debezium project because it relies on many technical components like Kafka, Java, and a myriad of connectors. Although a conference (French) on this topic kept me engaged last June, I haven’t yet had the opportunity to lead a large-scale project that justified such a significant human investment in its implementation.

There are numerous free or paid tools, each with its own purpose and philosophy. Choosing one over another depends on the migration teams and their technical maturity, but ultimately, the choice is free!