Developpement on Discovery notebook

Window functions to the rescue

Florent Jardin — Tue, 17 Dec 2024 14:00:00 +0200

PostgreSQL comes with a variety of functions that allow you to group rows into a “window” and perform calculations on that window. By using these functions, you can create more advanced and efficient queries for analyzing your database.

In early 2023, I contributed to a project that converts data models to PostgreSQL, called db_migrator. On this occasion, I (re)discovered the power of these window functions with the SQL language. In this article, I revisit a specific case of transforming the upper bounds of a partitioned table into an array of boundaries.

Through a window

We do not hide it, the use of a window in a query is (really) not usual. Windowing is a particularly useful technique for performing time-shifted data analysis, like cumulative totals, moving averages, trends, etc.

The most commonly used windowing functions in PostgreSQL are first_value(), last_value(), rank(), row_number(), and lag(). These functions can be combined with aggregation functions such as sum(), avg(), count(), min(), and max() to produce even more useful results.

Let us see a practical example with the first_value() method that presents the first line of the window attached to each line. With the concrete case of partitioning, we want to know the first partition defined for the table for each line of a table named partitions:

CREATE TABLE partitions (
 table_name text,
 partition_name text,
 upper_bound text,
 position int
);

INSERT INTO partitions VALUES
 ('tab', 'less_than_10', '10', 1),
 ('tab', 'less_than_20', '20', 2),
 ('tab', 'less_than_30', '30', 3),
 ('tab', 'less_than_40', '40', 4),
 ('tab', 'less_than_max', 'MAXVALUE', 5);

SELECT table_name, partition_name, upper_bound, position,
 first_value(partition_name) OVER (
 PARTITION BY table_name
 ORDER BY position
 ) AS first_partition
FROM partitions;

 table_name | partition_name | upper_bound | position | first_partition
------------+----------------+-------------+----------+-----------------
 tab | less_than_10 | 10 | 1 | less_than_10
 tab | less_than_20 | 20 | 2 | less_than_10
 tab | less_than_30 | 30 | 3 | less_than_10
 tab | less_than_40 | 40 | 4 | less_than_10
 tab | less_than_max | MAXVALUE | 5 | less_than_10

A window function is used in conjunction with the OVER clause to specify the window boundaries, determining column key on which the window function applies. Here, the window groups the partition data by the table name (table_name) by sorting them with the ORDER BY clause.

A window moves for each line, expanding its boundaries according to the new lines it reads in the declared order. If we want to retrieve the last partition defined for the table of each line, it is necessary to reverse the sorting of the window, in order to start reading from the bottom up to the current line and sort again the result.

SELECT table_name, partition_name, upper_bound, position,
 first_value(partition_name) OVER (
 PARTITION BY table_name
 ORDER BY position DESC
 ) AS last_partition
FROM partitions ORDER BY position;

 table_name | partition_name | upper_bound | position | last_partition
------------+----------------+-------------+----------+----------------
 tab | less_than_10 | 10 | 1 | less_than_max
 tab | less_than_20 | 20 | 2 | less_than_max
 tab | less_than_30 | 30 | 3 | less_than_max
 tab | less_than_40 | 40 | 4 | less_than_max
 tab | less_than_max | MAXVALUE | 5 | less_than_max

Looking for the lower bound

With Oracle or even MySQL, partitioning by RANGE only requires the upper bound to be defined to distribute the data into the partitions. Thus, creating a partitioned table is similar to the following example:

CREATE TABLE tab (
 id INT NOT NULL,
 junk TEXT NOT NULL
)
PARTITION BY RANGE (id) (
 PARTITION less_than_10 VALUES LESS THAN (10),
 PARTITION less_than_20 VALUES LESS THAN (20),
 PARTITION less_than_30 VALUES LESS THAN (30),
 PARTITION less_than_40 VALUES LESS THAN (40),
 PARTITION less_than_max VALUES LESS THAN MAXVALUE
);

PostgreSQL is slightly more strict on the definition of RANGE partitions with the use of lower and upper bounds, like this:

CREATE TABLE tab (
 id INT NOT NULL,
 junk TEXT NOT NULL
)
PARTITION BY RANGE (id);

CREATE TABLE less_than_10
 PARTITION OF tab FOR VALUES FROM (MINVALUE) TO (10);
CREATE TABLE less_than_20
 PARTITION OF tab FOR VALUES FROM (10) TO (20);
CREATE TABLE less_than_30
 PARTITION OF tab FOR VALUES FROM (20) TO (30);
CREATE TABLE less_than_40
 PARTITION OF tab FOR VALUES FROM (30) TO (40);
CREATE TABLE less_than_max
 PARTITION OF tab FOR VALUES FROM (40) TO (MAXVALUE);

We want to build an array of lower and upper bounds from multiple rows of our previously defined partitions table. The lag() function is the perfect candidate for this task, as it allows us to access the value of a previous row in the same window.

lag (value anycompatible [, offset integer [, default anycompatible ]])
 → anycompatible

Returns value evaluated at the row that is offset rows before the current row within the partition; if there is no such row, instead returns default (which must be of a type compatible with value).

The new query looks like this:

SELECT table_name, partition_name,
 ARRAY[
 lag(upper_bound, 1, 'MINVALUE') OVER (
 PARTITION BY table_name
 ORDER BY position
 ),
 upper_bound
 ]::text[] as boundaries
FROM partitions;

 table_name | partition_name | boundaries
------------+----------------+---------------
 tab | less_than_10 | {MINVALUE,10}
 tab | less_than_20 | {10,20}
 tab | less_than_30 | {20,30}
 tab | less_than_40 | {30,40}
 tab | less_than_max | {40,MAXVALUE}

Another query with first_value() had been used before lag() was proposed by Laurenz Albe in the final form of my work. For an equivalent result, it was considered less suitable by relying unnecessarily on more advanced clauses such as RANGE BETWEEN and EXCLUDE. I let you appreciate the full syntax.

SELECT table_name, partition_name,
 ARRAY[
 coalesce(first_value(upper_bound) OVER (
 PARTITION BY table_name ORDER BY position
 RANGE BETWEEN 1 PRECEDING AND CURRENT ROW
 EXCLUDE CURRENT ROW
 ), 'MINVALUE'),
 upper_bound
 ]::text[] as boundaries
FROM partitions;

Conclusion

Windowing is not a easy concept to grasp, especially since concrete and relevant cases are not encountered frequently. Without mastering its subtleties, it is useful to know its purpose, or at least its existence. It would have been tempting to make joins, CTEs, or subqueries but at the cost of reduced readability and potential optimization barriers.

Substituting a variable in a SQL script

Florent Jardin — Mon, 25 Nov 2024 09:00:00 +0100

In a world where we constantly seek to automate repetitive tasks, it is common to write down a query in a script, make it more convenient, and eventually integrate the whole thing into a project’s codebase. Tools like SQL*Plus and psql can be powerful allies in this game, as relevant as Bash or Python interpreters.

In several projects I have been involved in, I have come across a large number of those kinds of scripts. Some of them have the particularity of offering input parameters, processed by SQL*Plus with the very comfortable mechanism named variable substitution. In this article, I share some tips to convert them to an equivalent syntax that PostgreSQL’s psql tool can parse and manage.

Substituting a variable…

As a starting point, let’s look at the syntax supported by Oracle’s tool with a progressive example. We stand in the shoes of a user who wants to get the result of sales from a dummy company, applying two filters based on a product identifier and two dates passed as parameters.

-- sqlplus-report-01.sql
CONNECT user/password@database

DEF product_id = '&1'
DEF start_date = '&2'
DEF end_date = '&3'

SELECT NVL(SUM(amount), 0) AS total_amount
 FROM orders
 JOIN products USING (product_id)
 WHERE product_id = &product_id
 AND order_date BETWEEN TO_DATE('&start_date', 'YYYY-MM-DD')
 AND TO_DATE('&end_date', 'YYYY-MM-DD');

QUIT

$ sqlplus -S /nolog @sqlplus-report-01.sql 20 2024-11-01 2024-11-30
old 3: WHERE product_id = &product_id
new 3: WHERE product_id = 20
old 4: AND order_date BETWEEN TO_DATE('&start_date', 'YYYY-MM-DD')
new 4: AND order_date BETWEEN TO_DATE('2024-11-01', 'YYYY-MM-DD')
old 5: AND TO_DATE('&end_date', 'YYYY-MM-DD')
new 5: AND TO_DATE('2024-11-30', 'YYYY-MM-DD')

TOTAL_AMOUNT
------------
 1378.98

As shown above, the substitution takes place in SQL*Plus as soon as the & symbol is encountered, whether its content is surrounded by single quotes or not. At the beginning of the script, I applied a simple strategy by defining each variable with a meaningful name, to facilitate the readability and maintenance of the script. We don’t want to rely on variables named according to their position in the list of arguments.

psql can provide a similar feature, but the syntax is slightly different. We have to assign variables outside the script, using the -v or --set option. These variables will then be substituted in the script using the interpolation system.

-- psql-report-01.sql
\pset footer off

SELECT COALESCE(SUM(amount), 0) AS total_amount
 FROM orders
 JOIN products USING (product_id)
 WHERE product_id = :product_id
 AND order_date BETWEEN :'start_date'::date
 AND :'end_date'::date;

$ psql -f psql-report-01.sql \
$ -v product_id=20 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
 total_amount
--------------
 1378.98

From this point, we can see that the conversion is quite straightforward. The substitution syntax is similar with the : character instead of &. However, if the variable states as a literal string, we must use the : character outside the quotes and not inside.

… in an anonymous block

Things get complicated when the script becomes more complex and requires us to manipulate our previous variables in an anonymous PL/SQL block. Let’s take our script back to enhance it with a personalized message if the requested product does not exist. A first read on the products table should eventually raise the NO_DATA_FOUND exception that we want to catch.

-- sqlplus-report-02.sql
CONNECT user/password@database

DEF product_id = '&1'
DEF start_date = '&2'
DEF end_date = '&3'

SET serveroutput ON
SET feedback OFF
SET verify OFF

DECLARE
 p_id products.product_id%TYPE;
 p_sum NUMBER;
 p_start DATE := TO_DATE('&start_date', 'YYYY-MM-DD');
 p_end DATE := TO_DATE('&end_date', 'YYYY-MM-DD');

BEGIN
 SELECT product_id INTO p_id
 FROM products
 WHERE product_id = &product_id;

 SELECT NVL(SUM(amount), 0) INTO p_sum
 FROM orders
 WHERE product_id = p_id
 AND order_date BETWEEN p_start AND p_end;

 DBMS_OUTPUT.PUT_LINE('Total amount: ' || p_sum);

EXCEPTION
 WHEN NO_DATA_FOUND THEN
 DBMS_OUTPUT.PUT_LINE('Product ' || &product_id || ' does not exist');
END;
/

QUIT

$ sqlplus -S /nolog @sqlplus-report-02.sql 120 2024-11-01 2024-11-30
Product 120 does not exist

Unfortunately, our beloved psql command-line interface does not support the ubstitution of variables in an anonymous block and any attempt to do so will result in a syntax error.

-- psql-report-02-wrong.sql
DO $$
DECLARE
 p_id products.product_id%TYPE;
 p_sum numeric;
 p_start date := :'start_date'::date;
 p_end date := :'end_date'::date;

BEGIN
 SELECT product_id INTO STRICT p_id
 FROM products
 WHERE product_id = :product_id;

 SELECT COALESCE(SUM(amount), 0) INTO p_sum
 FROM orders
 WHERE product_id = p_id
 AND order_date BETWEEN p_start AND p_end;

 RAISE NOTICE 'Total amount: %', p_sum;

EXCEPTION
 WHEN no_data_found THEN
 RAISE NOTICE 'Product % does not exist', :product_id;
END;
$$;

$ psql -f psql-report-02-wrong.sql \
$ -v product_id=120 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
psql:psql-report-02-wrong.sql:25: ERROR: syntax error at or near ":"
LINE 5: p_start date := :'start_date'::date;
 ^

Do not panic! psql provides us with a way to achieve the same result, with a deeper rewrite of the script. We will use the \gset meta-command to store the product state and then the \if meta-command to perform the control.

-- psql-report-02.sql
\pset footer off

SELECT NOT EXISTS(
 SELECT product_id
 FROM products
 WHERE product_id = :product_id
) AS unknown_product \gset

\if :unknown_product
 \echo 'Product' :product_id 'does not exist'
 \quit
\endif

SELECT COALESCE(SUM(amount), 0) AS total_amount
 FROM orders
 JOIN products USING (product_id)
 WHERE product_id = :product_id
 AND order_date BETWEEN :'start_date'::date
 AND :'end_date'::date;

$ psql -f report.sql \
$ -v product_id=120 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
Product 120 does not exist

… at any costs

In the wild, it may happen that psql meta-commands are not enough to handle all the intricacies (and absurdities) that SQL*Plus-compatible scripts can present. Let’s move on with a more complex need we want to address.

Our user now wants to get a performance score for the sales period of his product, comparing it with the previous period. We would need a cursor here to reuse the same sales calculation query multiple times and a function to compute a score by handling the possible division by zero.

-- sqlplus-report-03.sql
CONNECT user/password@database

DEF product_id = '&1'
DEF start_date = '&2'
DEF end_date = '&3'

SET serveroutput ON
SET feedback OFF
SET verify OFF

DECLARE
 p_id products.product_id%TYPE;
 p_sum NUMBER;
 p_prev_sum NUMBER;
 p_start DATE := TO_DATE('&start_date', 'YYYY-MM-DD');
 p_end DATE := TO_DATE('&end_date', 'YYYY-MM-DD');
 p_prev_start DATE := p_start - (p_end - p_start);
 p_prev_end DATE := p_start - 1;

 CURSOR c_orders (v_start DATE, v_end DATE) IS
 SELECT SUM(amount) AS total_amount
 FROM orders
 WHERE product_id = p_id
 AND order_date BETWEEN v_start AND v_end;

 FUNCTION score(p_sum NUMBER, p_prev_sum NUMBER)
 RETURN NUMBER IS
 v_score NUMBER;
 BEGIN
 RETURN ROUND((p_sum - p_prev_sum) / p_prev_sum * 100, 2);
 EXCEPTION
 WHEN ZERO_DIVIDE THEN
 RETURN NULL;
 END;

BEGIN
 SELECT product_id INTO p_id
 FROM products
 WHERE product_id = &product_id;

 OPEN c_orders(p_start, p_end);
 FETCH c_orders INTO p_sum;
 CLOSE c_orders;

 OPEN c_orders(p_prev_start, p_prev_end);
 FETCH c_orders INTO p_prev_sum;
 CLOSE c_orders;

 DBMS_OUTPUT.PUT_LINE('Total amount: ' || p_sum);
 DBMS_OUTPUT.PUT_LINE('Performance score: ' || score(p_sum, p_prev_sum));

EXCEPTION
 WHEN NO_DATA_FOUND THEN
 DBMS_OUTPUT.PUT_LINE('Product ' || &product_id || ' does not exist');
END;
/

QUIT

$ sqlplus -S /nolog @sqlplus-report-03.sql 20 2024-11-01 2024-11-30
Total amount: 1378.98
Performance score: 162.99

Of course, the example is deliberately simplistic and could be addressed by a window function, but we will play the game of converting the script as close as possible to its original syntax. To do this, let’s identify the weaknesses of PL/pgSQL.

Temporary routines are not supported

PL/pgSQL language does not allow the declaration of stored functions or storeed procedures inside an anonymous block. The only alternative is to define it globally, so that it is known to the parser and executed correctly. In this present case, I do not want the function to persist after the call of my script. No problem, we can create a temporary function in the pg_temp schema!

-- temp-function.sql
DO $$
BEGIN
 CREATE FUNCTION pg_temp.score(p_sum numeric, p_prev_sum numeric)
 RETURNS numeric LANGUAGE plpgsql AS $func$
 BEGIN
 RETURN ROUND((p_sum - p_prev_sum) / p_prev_sum * 100, 2);
 EXCEPTION
 WHEN division_by_zero THEN
 RETURN NULL;
 END;
 $func$;

 RAISE NOTICE 'Score: %', pg_temp.score(100.0, 50.0);
 RAISE NOTICE 'Score: %', pg_temp.score(100.0, 0);
END;
$$;

$ psql -f temp-function.sql
psql:temp-function.sql:16: NOTICE: Score: 100.00
psql:temp-function.sql:16: NOTICE: Score: <NULL>
DO

The pg_temp.score(numeric, numeric) function is now accessible within the block, and only for the duration of the session. Exception handling respects the expressed need. We just had to replace ZERO_DIVIDE with division_by_zero as describe in the documentation.

SQL substitution is not supported

Our main issue strikes back: substitution does not occur when we are in a PL/pgSQL block. To work around this limitation, we will have to use the custom session variables made available for the PostgreSQL extension ecosystem.

Those variables can be manipulated as well in psql with the SET keyword as in SQL or PL/pgSQL with the current_setting and set_config functions. The only constraint is to define a prefix that does not conflict with that of other variables already declared in our instance.

It means that our parameters can be set as session variables in the very beginning of the script. As soon as we are in an anonymous block, we must then assign them back to regular variables as shown in the following code:

SET my.product_id = :product_id;
SET my.start_date = :start_date;
SET my.end_date = :end_date;

DO $$
DECLARE
 my_id int := current_setting('my.product_id')::int;
 p_start date := current_setting('my.start_date')::date;
 p_end date := current_setting('my.end_date')::date;
BEGIN
 ...
END;
$$;

Full conversion

A new version of the script, converted from PL/SQL to PL/pgSQL, can be obtained with all previous tricks put together:

-- psql-report-03.sql
\set QUIET on

SET my.product_id = :product_id;
SET my.start_date = :'start_date';
SET my.end_date = :'end_date';

DO $$
DECLARE
 my_id int := current_setting('my.product_id')::int;
 p_id products.product_id%TYPE;
 p_sum numeric;
 p_prev_sum numeric;
 p_start date := current_setting('my.start_date')::date;
 p_end date := current_setting('my.end_date')::date;
 p_prev_start date := p_start - interval '1 day' * (p_end - p_start);
 p_prev_end date := p_start - interval '1 day';

 c_orders CURSOR (v_start date, v_end date) IS
 SELECT COALESCE(SUM(amount), 0)
 FROM orders
 WHERE product_id = p_id
 AND order_date BETWEEN v_start AND v_end;

BEGIN
 CREATE FUNCTION pg_temp.score(p_sum numeric, p_prev_sum numeric)
 RETURNS numeric LANGUAGE plpgsql AS $func$
 BEGIN
 RETURN ROUND((p_sum - p_prev_sum) / p_prev_sum * 100, 2);
 EXCEPTION
 WHEN division_by_zero THEN
 RETURN NULL;
 END;
 $func$;

 SELECT product_id INTO STRICT p_id
 FROM products
 WHERE product_id = my_id;

 OPEN c_orders(p_start, p_end);
 FETCH c_orders INTO p_sum;
 CLOSE c_orders;

 OPEN c_orders(p_prev_start, p_prev_end);
 FETCH c_orders INTO p_prev_sum;
 CLOSE c_orders;

 RAISE NOTICE 'Total amount: %', p_sum;
 RAISE NOTICE 'Performance score: %', pg_temp.score(p_sum, p_prev_sum);

EXCEPTION
 WHEN no_data_found THEN
 RAISE NOTICE 'Product % does not exist', my_id;
END;
$$;

$ psql -f psql-report-03.sql\
$ -v product_id=20 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
psql:psql-report-03.sql:55: NOTICE: Total amount: 1378.98
psql:psql-report-03.sql:55: NOTICE: Performance score: 162.99

Conclusion

This kind of rewriting becomes affordable when we know the right tricks, without losing sight of the initial goal. Of course, a one-to-one translation, by preserving the syntax and the original intent, could be more challenging than that blog post example.

I’m a bit skeptical about those who want to keep such development practices. They seem outdated to me. Isn’t migrating to PostgreSQL an opportunity to modernize our codebase or to question our relationship with technology? In our example, as I previously mentioned, it is possible to get the same result in a single SQL, combining meta-commands, variables substitution, common-table expressions, and a the LAG window function.

In that way, we throw away the PL/pgSQL block and its limitations.

-- psql-report-04.sql
\pset footer off

SELECT NOT EXISTS(
 SELECT product_id
 FROM products
 WHERE product_id = :product_id
) AS unknown_product \gset

\if :unknown_product
 \echo 'Product' :product_id 'does not exist'
 \quit
\endif

SELECT :'end_date'::date - :'start_date'::date AS days \gset
SELECT :'start_date'::date - :days AS prev_start \gset

WITH periods AS (
 SELECT d AS start_date, d + :days * '1 day'::interval AS end_date
 FROM generate_series(:'prev_start'::date,
 :'start_date'::date,
 :days * '1 day'::interval) AS s(d)
), amounts AS (
 SELECT start_date, COALESCE(SUM(amount), 0) AS total_amount,
 LAG(SUM(amount), 1) OVER (ORDER BY start_date) AS prev_total_amount
 FROM orders
 JOIN periods ON order_date BETWEEN start_date AND end_date
 WHERE product_id = :product_id
 GROUP BY start_date
)
SELECT total_amount,
 ROUND((total_amount - prev_total_amount) /
 prev_total_amount * 100, 2) AS performance_score
 FROM amounts
 WHERE prev_total_amount IS NOT NULL;

$ psql -f psql-report-04.sql\
$ -v product_id=20 \
$ -v start_date='2024-11-01' -v end_date='2024-11-30'
 total_amount | performance_score
--------------+-------------------
 1378.98 | 162.99

You can find all the scripts of this article, as well as the DDL commands to the data model, at this address.

Hierarchical data types

Florent Jardin — Thu, 19 Sep 2024 13:20:00 +0200

The SQL standard defines a set of rules so that database systems can be interchangeable, but there are small singularities in the wild. In this regard, the hierarchyid data type provided by SQL Server is a striking example. If you are switching to PostgreSQL, two solutions are available to you.

A first and simpler solution consists in linking each node to its parent using a new parentid column and applying a foreign key constraint. Another, more complete approach consists in using the ltree extension. This article deals with the latter case.

Searching for descendants

The hierarchyid type is designed to represent a hierarchical relationship as a binary tree. It stores the list of successive nodes up to the root in a single column. Thus, the node /1/1/2/ represents a level 3 node, child of the node /1/1/, itself child of the node /1/, itself child of the root /.

Many built-in methods are part of the Transact-SQL language to manipulate the data. They all traverse the binary tree to quickly extract the desired information without having to join the nodes of the table together.

ToString() : returns the textual representation of the current node.
GetLevel() : returns the depth level of the current node.
GetAncestor(n) : returns the n-level node in the tree.
GetDescendant(c1, c2) : returns a new child node between two nodes.

PostgreSQL brings us a new data type, ltree, to store hierarchical data. It is a complex data type that can store labels up to 1,000 characters long, separated by dots. The path can contain up to 65,635 labels.

CREATE EXTENSION ltree;

CREATE TABLE locations (
 id ltree PRIMARY KEY,
 location text NOT NULL,
 locationtype text NOT NULL
);

INSERT INTO locations VALUES
 ('1', 'Earth', 'Planet'),
 ('1.1', 'Europe', 'Continent'),
 ('1.1.1', 'France', 'Country'),
 ('1.1.1.1', 'Paris', 'City'),
 ('1.1.2', 'Spain', 'Country'),
 ('1.1.2.1', 'Madrid', 'City'),
 ('1.2', 'South-America', 'Continent'),
 ('1.2.1', 'Brazil', 'Country'),
 ('1.2.1.1', 'Brasilia', 'City'),
 ('1.2.2', 'Bahia', 'State'),
 ('1.2.2.1', 'Salvador', 'City'),
 ('1.3', 'Antarctica', 'Continent'),
 ('1.3.1', 'McMurdo Station', 'City');

Looking for the depth level of a node becomes trivial with the nlevel() function provided by the ltree extension.

SELECT id, location, locationtype, nlevel(id) AS level
 FROM locations ORDER BY id;

 id | location | locationtype | level
---------+-----------------+--------------+-------
 1 | Earth | Planet | 1
 1.1 | Europe | Continent | 2
 1.1.1 | France | Country | 3
 1.1.1.1 | Paris | City | 4
 1.1.2 | Spain | Country | 3
 1.1.2.1 | Madrid | City | 4
 1.2 | South-America | Continent | 2
 1.2.1 | Brazil | Country | 3
 1.2.1.1 | Brasilia | City | 4
 1.2.2 | Bahia | State | 3
 1.2.2.1 | Salvador | City | 4
 1.3 | Antarctica | Continent | 2
 1.3.1 | McMurdo Station | City | 3
 2.1.1 | unknown | State | 3
(14 rows)

Another method named subpath() allows you to retrieve a part of the path of a node. Let’s see how to get the parent node of each node in the table.

SELECT id, location, locationtype, subpath(id, 0, nlevel(id) - 1) AS parentid
 FROM locations ORDER BY id;

 id | location | locationtype | parent
---------+-----------------+--------------+--------
 1 | Earth | Planet |
 1.1 | Europe | Continent | 1
 1.1.1 | France | Country | 1.1
 1.1.1.1 | Paris | City | 1.1.1
 1.1.2 | Spain | Country | 1.1
 1.1.2.1 | Madrid | City | 1.1.2
 1.2 | South-America | Continent | 1
 1.2.1 | Brazil | Country | 1.2
 1.2.1.1 | Brasilia | City | 1.2.1
 1.2.2 | Bahia | State | 1.2
 1.2.2.1 | Salvador | City | 1.2.2
 1.3 | Antarctica | Continent | 1
 1.3.1 | McMurdo Station | City | 1.3
 2.1.1 | unknown | State | 2.1
(14 rows)

Least but not last, the search for child nodes from a given node is possible thanks to specialized comparison operators. To improve performance, it is recommended to create a GIST index on the primary key column id.

CREATE INDEX ON locations USING GIST (id);

All cities in Europe can be retrieved with the following query.

SELECT l1.*
 FROM locations l1
 JOIN locations l2 ON l1.id <@ l2.id
 WHERE l1.locationtype = 'City' AND l2.location = 'Europe';

 id | location | locationtype
---------+----------+--------------
 1.1.1.1 | Paris | City
 1.1.2.1 | Madrid | City
(2 rows)

Working under constraints

The declared primary key constraint prevents me from inserting an existing path. However, it is still possible to add a new row whose path does not have any ancestor in the table.

INSERT INTO locations VALUES ('2.1.1', 'Unknown', 'Continent');

INSERT 0 1

Things does not change much between SQL Server and PostgreSQL, it is necessary to add an additional column, named parentid for instance, to enforce the foreign key constraint. The following query reuses the subpath() function ensuring a null value is inserted if it is a root node.

DELETE FROM locations WHERE id <@ '2';

ALTER TABLE locations ADD COLUMN parentid ltree
 REFERENCES locations (id)
 GENERATED ALWAYS AS (
 CASE subpath(id, 0, nlevel(id) - 1)
 WHEN '' THEN null
 ELSE subpath(id, 0, nlevel(id) - 1)
 END
 ) STORED;

By now, whenever a new row is inserted, the foreign key constraint is automatically checked.

INSERT INTO locations VALUES ('2.1.1', 'Unknown', 'Continent');

ERROR: insert or update on table "locations" violates
 foreign key constraint "locations_parentid_fkey"
DETAIL: Key (parentid)=(2.1) is not present in table "locations".

Just one solution among others

This kind of data type is a clever solution to store hierarchical data and is easily available with PostgreSQL. However, it is an extension of the SQL language and each database engine can propose an implementation that can radically change from one system to another.

As I mentioned in the introduction, the universal answer is to rebuild a hierarchical relationship using a recursive query and the WITH RECURSIVE syntax. To take the example of the locations table, the list of cities in Europe could be obtained as follows.

WITH RECURSIVE loc AS (
 SELECT id, parentid, location, locationtype
 FROM locations
 WHERE location = 'Europe'
 UNION ALL
 SELECT l.id, l.parentid, l.location, l.locationtype
 FROM locations l
 JOIN loc r ON l.parentid = r.id
)
SELECT * FROM loc
 WHERE locationtype = 'City';

 id | parentid | location | locationtype
----+----------+----------+--------------
 4 | 3 | Paris | City
 6 | 5 | Madrid | City
(2 rows)

An assistant to copy data from a remote server

Florent Jardin — Tue, 28 May 2024 00:00:00 +0000

During the last PGSession organized by Dalibo, I wrote and led a workshop (french) on the migration to PostgreSQL using Foreign Data Wrappers, or FDW. This was an opportunity to present to the general public the db_migrator extension for which I wrote an article on this blog.

While working on this workshop, we noticed that copying data with the db_migrator extension is not perfectly supported. Indeed, although there is a low-level function to distribute the transfer table by table over several processes, many situations will require writing a large number of SQL queries to get out of trouble. Over the following months, I worked on the design of an assistant written in PL/pgSQL whose purpose is to simplify the generation of these queries.

Data transfer without an assistant

There is no magic in the operation of copying data through a Foreign Data Wrapper. Everything boils down to a series of INSERT queries that must be executed in a predefined order. Let’s take the 16 tables of the well-known “Sakila” data model (available at this address) to illustrate the need for generating transfer queries.

For the purpose of this article, I have a PostgreSQL database with the two following schemas that I will use in my demonstrations:

mysql: the source schema containing the definition of the external tables via the mysql_fdw extension;
public: the target schema where the data will be copied.

Each external table is subject to a quick study to obtain the correct column type match, and their definition is kept in an SQL file for the project teams. For example, the rental foreign tables is defined as follows:

CREATE FOREIGN TABLE mysql.rental (
 rental_id integer NOT NULL,
 rental_date timestamp without time zone NOT NULL,
 inventory_id integer NOT NULL,
 customer_id smallint NOT NULL,
 return_date timestamp without time zone,
 staff_id smallint NOT NULL,
 last_update timestamp without time zone NOT NULL
)
SERVER sakila_mysql
OPTIONS (
 dbname 'sakila',
 table_name 'rental'
);

When creating the public.rental table that will host the data, it is opportune to decide whether we want to set up partitioning, something that db_migrator is able to identify and set up. For the example, I take back the strict structure using the CREATE TABLE LIKE syntax to create all my target tables.

CREATE TABLE public.actor (LIKE mysql.actor);
CREATE TABLE public.address (LIKE mysql.address);
...

Even before setting up an INSERT query generator, it is easy to see the look of these queries. Each line of the external table will be read through a global SELECT, then inserted into the target table. The migration script contains 16 instructions, one for each table.

-- insert.sql
INSERT INTO public.actor SELECT * FROM mysql.actor;
INSERT INTO public.address SELECT * FROM mysql.address;
...
INSERT INTO public.store SELECT * FROM mysql.store;

To benefit from several processes, I appreciate the xargs tool which allows distributing each line of the insert.sql file on a new psql session. This technique was presented in the February workshop, especially to parallelize the construction of indexes and primary keys, defined in an SQL file.

$ xargs -P 4 -a insert.sql -d '\n' -I % sh -c 'psql -c "%"'
INSERT 0 16
INSERT 0 603
...
INSERT 0 16044

This method is not perfect, it lacks verbosity, the SQL queries are static. Anyway, let’s see the next part to discover what my assistant can bring.

Demonstration of the assistant

Unlike my other PL/pgSQL projects, this assistant is not an extension and must be installed as a simple script. Once downloaded, it is sufficient to invoke it on the database of your choice with the following command:

$ psql -d sakila -f fdw-assistant.sql

The default schema is named assistant and contains a configuration table simply called config. For each table to be migrated, one must insert a single line that will serve as the starting point for generating data migration queries. In the current version, we find the following parameters:

source: the external table containing the data to be copied;
target: the target table where the data will be copied;
pkey: the primary key column of the target table;
priority: the lowest values define the tables to be processed first;
parts: the number of processes to launch for copying the data;
trunc: an option to empty the target table before copying the data;
condition: a WHERE clause to filter the data to be copied;
batchsize: the number of rows to copy before performing an intermediate COMMIT.

To initialize this table in the first instance, it is necessary to know at least the primary key columns of each remote table. By reporting the information from the relational diagram of the Sakila database, we can fill the config table as follows:

INSERT INTO assistant.config (source, target, pkey)
VALUES
 ('mysql.actor', 'public.actor', 'actor_id'),
 ('mysql.address', 'public.address', 'address_id'),
 ...
 ('mysql.store', 'public.store', 'store_id');

For each transfer, we indicate the source and target table, as well as the primary key column. The latter is required to sort the rows, split the transfer into several batches, and restart the transfer in case of interruption.

With this configuration, we can move on to the planning step. The stage and job tables are fed with new elements that will be used to drive and track the different transfers to be done.

SELECT * FROM assistant.plan();

 target | invocation
---------------+--------------------------
 customer | CALL assistant.copy(1);
 address | CALL assistant.copy(2);
 ...
(16 rows)

A view named report allows you to follow the progress of the different steps by joining the stage and job tables. It provides very useful elements to monitor the progress and throughput of the transfers.

SELECT target, state, rows FROM assistant.report WHERE stage_id = 1;

 target | state | rows
---------------+---------+------
 rental | pending | 0
 actor | pending | 0
 ...
(16 rows)

Rows returned by the plan() command can then be invoked one after the other with the \gexec meta-command of psql, or by using the file technique and distributing the queries with xargs.

Calling the copy() method is responsible for building the INSERT statement for copying data from a remote table to a local table. For example, for the customer table, the result of the call will be as follows:

CALL assistant.copy(1);

NOTICE: Executing: TRUNCATE public.customer
NOTICE: Executing: SELECT count(customer_id) FROM mysql.customer
 WHERE customer_id > 0
NOTICE: Executing: INSERT INTO public.customer
 SELECT customer_id, store_id, first_name, last_name, email,
 address_id, active, create_date, last_update
 FROM mysql.customer
 WHERE customer_id > 0 ORDER BY customer_id
CALL

At the end of the transfer, the report view provides a summary of the operation.

SELECT * FROM assistant.report
 WHERE stage_id = 1 AND target = 'public.customer'::regclass;

-[ RECORD 1 ]-------------------------
stage_id | 1
target | public.customer
job_start | 2024-05-28 10:19:18.334917
state | completed
rows | 599
total | 599
elapsed | 00:00:00.081273
rate | 7370.22
progress | 1.00
eti | 00:00:00
eta | 2024-05-28 10:19:18.334917

Distribution over several processes

As the assistant was designed, I felt the need to enrich the underlying queries to respond to other recurring use cases in the data migration domain. Among these, we find the ability to distribute the rows of the same table over several sessions, each with a WHERE clause based on the result of the Euclidean division (modulo) of the primary key.

To activate this feature, simply fill in the parts parameter in the config table. For example, for the film table, we can define:

UPDATE assistant.config SET parts = 4 WHERE target = 'public.film'::regclass;

This configuration takes effect during the next planning. The call to the plan() method then inserts four lines into the job table for the film table, each attached to a value between 0 and 3. The condition column is then enriched for the rows of the task table in view of the next step.

SELECT invocation FROM assistant.plan('{public.film}');

 invocation
--------------------------
 CALL assistant.copy(17);
 CALL assistant.copy(18);
 CALL assistant.copy(19);
 CALL assistant.copy(20);
(4 rows)

SELECT job_id, job.target, state, part, condition
 FROM assistant.job JOIN assistant.task USING (job_id)
 WHERE stage_id = 2;

 job_id | target | state | part | condition
--------+--------+---------+------+-----------------
 17 | film | pending | 0 | film_id % 4 = 0
 18 | film | pending | 1 | film_id % 4 = 1
 19 | film | pending | 2 | film_id % 4 = 2
 20 | film | pending | 3 | film_id % 4 = 3

When calling the copy() method, the assistant builds the INSERT queries based on the conditions previously defined. For example, for the first part of the film table, the trace indicates the INSERT queries that were generated.

CALL assistant.copy(17);

NOTICE: Executing: TRUNCATE public.film
NOTICE: Executing: SELECT count(film_id) FROM mysql.film
 WHERE film_id > 0 AND film_id % 4 = 0
NOTICE: Executing: INSERT INTO public.film
 SELECT film_id, title, description, release_year,
 language_id, original_language_id, rental_duration,
 rental_rate, length, replacement_cost, rating,
 special_features, last_update
 FROM mysql.film WHERE film_id > 0 AND film_id % 4 = 0
 ORDER BY film_id
CALL

The TRUNCATE operation only occurs for the session whose value of part is equal to 0. In the nominal case, this session is launched before all the others to respect the expected behavior in the configuration with the trunc column (true by default).

The obvious interest of this method is to obtain the best insertion rate for a given table, relying on the extraction power of the remote server and the writing capacity of the local server. The rate can be consulted with the report view, especially to compare two loads for the same table, like the example of the film table. The rate column is expressed in number of rows per second.

SELECT stage_id, target, state, rate FROM assistant.report
 WHERE target = 'public.film'::regclass;

 stage_id | target | state | rate
----------+-------------+-----------+----------
 1 | public.film | completed | 19162.59
 2 | public.film | completed | 51389.37

Restart after interruption

The ability to restart a copy in batch mode in case of interruption is one of the reasons why the primary key must be filled in the configuration. The assistant relies on the last extracted primary key value (using a RETURNING clause) to know the next restart point. In designing this feature, I had to arbitrate the limitations imposed by this mechanism.

Composite primary keys are not allowed, as the RETURNING clause can only return a single value;
Tables with composite primary keys cannot benefit from batch processing, and therefore cannot be restarted after interruption;
Data is systematically sorted during extraction, even if batch processing is not enabled;
The primary key column must be of numeric type.

The activation of a batch processing consists of updating the batchsize column of the configuration table. Let’s take the example of the rental table:

UPDATE assistant.config SET batchsize = 1000
 WHERE target = 'public.rental'::regclass;

SELECT invocation FROM assistant.plan('{public.rental}');

 invocation
--------------------------
 CALL assistant.copy(21);
(1 row)

The data transfer from the rental table is then divided into batches of 1000 rows. It is of course possible to combine this technique with parallelization, the WHERE clause will do most of the work of repartition to prevent the same line from being exported twice.

CALL assistant.copy(21);

TRUNCATE public.rental
SELECT count(rental_id) FROM mysql.rental WHERE rental_id > 0
INSERT INTO public.rental SELECT rental_id, ...
 FROM mysql.rental WHERE rental_id > 0 ORDER BY rental_id LIMIT 1000
...
...
INSERT INTO public.rental SELECT rental_id, ...
 FROM mysql.rental WHERE rental_id > 16005 ORDER BY rental_id LIMIT 1000
INSERT INTO public.rental SELECT rental_id, ...
 FROM mysql.rental WHERE rental_id > 16049 ORDER BY rental_id LIMIT 1000
CALL

As soon as a query no longer returns any rows, the assistant considers that the transfer is complete. The job table is updated at each iteration to follow the last value of the primary key sequence.

SELECT target, lastseq FROM assistant.job
 WHERE stage_id = 3 AND target = 'public.rental'::regclass;

 target | lastseq
---------------+---------
 public.rental | 16049

In case of interruption, it is possible to restart the transfer by calling the copy() method with the same task identifier. The assistant takes care of resuming the transfer from the last known primary key value.

CALL assistant.copy(21);

NOTICE: Executing: SELECT count(rental_id) FROM mysql.rental
 WHERE rental_id > 16049
NOTICE: Executing: INSERT INTO public.rental SELECT rental_id, ...
 FROM mysql.rental WHERE rental_id > 16049
 ORDER BY rental_id LIMIT 1000
CALL

Conclusion

The design of such a tool was a small personal challenge in line with my research on migration to PostgreSQL with the exclusive help of Foreign Data Wrappers. My main source of inspiration remains the Ora2Pg project, one of the most advanced open-source tools to date in the field of migration.

I am aware of the technical limitations of this assistant, and the tinkering that remains to be done to make life easier for a consultant like me. In another article, I would like to present another tool called dispatch that I have been maintaining for some time and with which I answer questions about orchestration and traceability of migration steps.

By taking a step back, the basic concepts are there, any other tool in other languages could perfectly emerge and enrich the open-source ecosystem in the quest for migration to PostgreSQL.

On the road to freedom with db_migrator

Florent Jardin — Fri, 28 Jul 2023 00:00:00 +0000

Over the past months, I have spent several weeks contributing to the db_migrator extension. Written solely in PL/pgSQL, it enables the migration of schemas and data from a database system to PostgreSQL using the external data I had presented few years ago.

In this post, I present the functionality of the tool, its philosophy, and the reason I found for its existence, even though it joins the ecosystem of well-established open-source projects in the migration landscape. How does it compare to Ora2Pg or pgloader in terms of value and capabilities?

db_migrator enters the arena

My interest in this project dates back to last December when a colleague from Dalibo left us a similar tool, which allowed copying data from Oracle or Sybase instances using Foreign Data Wrappers (FDW) technology. Although this tool remained in alpha, many good ideas were experimented with internally.

The promise of FDWs lies in adhering to the SQL/MED standard, allowing a PostgreSQL instance to interface with another storage system and manipulate its data through external tables using simple SQL queries. Therefore, provided that a community has developed the wrapper, it becomes possible to query a remote catalog, replicate the structure of tables, their relationships, and constraints, and retrieve data into PostgreSQL.

And db_migrator enters the arena.

Made public in November 2019 by Laurenz Albe, well-known for his active contributions to PostgreSQL for decades and also for developing oracle_fdw, the extension presents itself as a generic tool where one must use plugins for FDW support. It is easy to create new plugins, as I discovered with the mysql_migrator plugin, written in just a few days, thanks to the comprehensive documentation of the API for plugins.

After installing the extensions with make install and the appropriate FDW for the system, it is necessary to create the objects in the database that will hold the future schemas and their data.

CREATE EXTENSION mysql_fdw;
CREATE EXTENSION mysql_migrator CASCADE;

CREATE SERVER mysql FOREIGN DATA WRAPPER mysql_fdw
 OPTIONS (host 'mysql_db', fetch_size '1000');
CREATE USER MAPPING FOR PUBLIC SERVER mysql
 OPTIONS (username 'root', password 'password');

The migration process can be performed in a single command for the simplest cases (no stored procedures or exotic column types) using the db_migrate() method. Otherwise, for more complex scenarios requiring adjustments such as changing column types or removing a table in the target schema, the migration may involve multiple steps.

During the development of the mysql_migration extension, I started with the sample database Sakila provided by MySQL to have comprehensive complexity. The first step involves creating two internal schemas, one with external tables provided by the plugin and the other with catalog tables that can be edited before the extension continues the migration.

SELECT db_migrate_prepare(
 plugin => 'mysql_migrator',
 server => 'mysql',
 only_schemas => '{sakila}'
);

This part can be relatively lengthy, as it involves retrieving the data model, which I refer to as the catalog, in the form of several tables that describe the structure of tables, column names, and associated constraints. The extension also imports the sources of all stored procedures, functions, and views but does not perform their conversion to PL/pgSQL (you cannot imagine the amount of work involved).

For the migration of the Sakila database, several modifications to the catalog are necessary. Like the rest of this extension, all the preparation is done in SQL, making it easy to automate with a single script serving as configuration.

/* exclude bytea columns from migration */
DELETE FROM pgsql_stage.columns WHERE type_name = 'bytea';

/* quote character expression */
UPDATE pgsql_stage.columns
 SET default_value = quote_literal(default_value)
 WHERE NOT regexp_like(default_value, '^\-?[0-9]+$')
 AND default_value <> 'CURRENT_TIMESTAMP';

/* disable view migration */
UPDATE pgsql_stage.views SET migrate = false;

Of course, we could go further, such as reinjecting the definition of rewritten views into the pgsql_stage.views table or enabling the migration of procedures by changing the migrate column of the pgsql_stage.functions table. However, let’s proceed with the next step.

SELECT db_migrate_mkforeign(
 plugin => 'mysql_migrator',
 server => 'mysql'
);

SELECT db_migrate_tables(
 plugin => 'mysql_migrator'
);

The first method, db_migrate_mkforeign(), is responsible for creating schemas and sequences, followed by foreign tables with columns based on the previous adjustments. Next comes the most crucial step, where we execute the function db_migrate_tables(): blank tables are created with their partitions if necessary, and for each of them, the data copying begins using the INSERT INTO SELECT * statement.

Other objects, such as indexes or constraints, have their own methods. It is necessary to create the functions before these objects if you encounter functional indexes or other similar cases.

SELECT db_migrate_functions(plugin => 'mysql_migrator');
SELECT db_migrate_triggers(plugin => 'mysql_migrator');
SELECT db_migrate_views(plugin => 'mysql_migrator');
SELECT db_migrate_indexes(plugin => 'mysql_migrator');
SELECT db_migrate_constraints(plugin => 'mysql_migrator');

It is possible that this mechanism may change in the future, especially if I manage to realize this issue, which would allow breaking down the db_migrate_*() methods into smaller steps.

The end of the migration process involves deleting the temporary schemas that contained the catalog tables.

SELECT db_migrate_finish();

One more migration tool

As I mentioned in the introduction, it is quite surprising to see a new migration tool emerge in 2023 (the version 1.0.0 was released in January with my patch on partitioning). In the open-source landscape, we can mention Ora2Pg, which released its version 24.0 in July with SQL Server support, and pgloader, which has an excellent reputation.

A vast number of projects are listed on the community wiki. Some are specialized for a single system, while others support migration for multiple systems. The majority of these projects are either proprietary or lack recent contributions. Many of them are black boxes, and their documentation may appear cryptic or almost non-existent.

The ecosystem is rich, and I do not claim to know all of its aspects, but I have had an intuition that I have been forming over the past few years. The global economy is in a state of turmoil. Some companies are doing well, while others are making budget cuts. The transition to a free and non-commercially licensed system like PostgreSQL remains relevant, perhaps even more urgent today compared to the past decade.

And yet, with my DBA perspective, I am not fully satisfied with the existing tools. I wish for a new alternative, something universal and accessible to everyone. If I turn to db_migrator today, it would be for the following main advantages:

A low-level implementation close to the instance: using PL/pgSQL as the exclusive language. This would not have been possible without the prolific development of Foreign Data Wrappers for a wide range of systems;
A high level of configuration flexibility: adjustments are made with UPDATE or DELETE queries on the catalog. Once one is familiar with the model of the catalog, it becomes easy to change behavior without consulting technical documentation on the available options;
Freedom in orchestration: currently, executions are triggered sequentially for indexes and constraints, but the tool’s architecture could allow external tools to consume the extension’s results and trigger operations in parallel;
Plugins have the freedom to enrich migration: if an operation is not generic, it is entirely possible to provide an additional method through the plugin. For example, the incremental copy (and its replication functions) in the ora_migrator plugin or the conversion of auto-increments to identity columns with the mysql_migrator plugin.

The road to freedom still seems long to achieve half of what Ora2Pg already offers, especially when it comes to automatic conversion, which is not on the agenda at all. But with small, regular, and thoughtful advancements, who knows?