Opensource on Discovery notebook

How to keep a community alive

Florent Jardin — Tue, 30 Jul 2024 09:30:00 +0000

The PG Day France took place on June 11th and 12th in Lille, my hometown. It is the event of the French PostgreSQL community that settles in a different city each year. The opportunity was too good for me and I met many people from all over France and its surroundings, to discuss PostgreSQL during two days of workshops and conferences.

For this edition, I had the pleasure of speaking and sharing my experience on the animation of the local Meetup group that I took over four years ago. In this article, I want to write down the main points discussed during this presentation, while waiting for the video of the conference to be posted online.

The slides of my presentation are available at this address (french).

What is a PUG?

A « PUG » is a PostgreSQL User Group, a community of PostgreSQL users who meet regularly to exchange about PostgreSQL. These local communities are common, especially for free software, and allow users to share their experiences and knowledge about the software.

Lille is a city with a rich economic ecosystem, and many communities are active in the region, such as the Ch’ti JUG (Java User Group), the GDG Lille (Google Developer Group), Nord Agile or Software Craft Lille. What could be more natural than wanting to offer a handful of this audience more meetings with PostgreSQL?

A dedicated page of the PostgreSQL project lists the different PUGs around the world, and it is possible to find groups in many countries, including France. At the time of writing my presentation, I counted 62 global groups, including 5 in France: Paris, Lyon, Nantes (not referenced), Toulouse and Lille.

Meetup’s platform is often used to organize meetings, and as proof, 46 PUGs are affiliated with it. This has been a convenience for more than 10 years, and most French groups are registered there.

The creation of a PUG recognized by the international community is a simple process, which requires to form an organizing committee and to apply to the email address usergroups@postgresql.org. The group must respect a number of rules, provided by the PUG policy, and must commit to respecting the values of the PostgreSQL community. In essence, here are the points to remember:

The group must be open to all, without discrimination.
Meetings must be proposed at least once every two years.
Meetings must not be subject to a non-disclosure agreement (NDA).
Meetings are attached to the geographical area of the group.
A company cannot be represented at 50% or more in the organizing committee.
The selection of conferences is at the discretion of the organizing committee.
Companies can promote their products and services, if their activities facilitate the adoption of PostgreSQL and if the content presented is technical in nature.
The group must disclose the names of sponsors, and may mention them in the introduction of meetings.
The PUG must adopt a code of conduct and can use the one from the PostgreSQL community.

Genesis of the group

The Meetup PostgreSQL Lille group was founded on February 25, 2016 on the eponymous platform. At the time, the idea of reproducing the Paris’s Meetup format was shared between Guillaume Lelarge and Pierre Hilbert, two Lille residents who regularly crossed paths at events. One of the main motivations was to promote free software with a sharing of feedback to inspire other local actors.

At that time, I myself attended events organized by the Meetup Oracle Paris and Province group, and I followed with interest the conferences about PostgreSQL in my region. The first meeting of the PG Lille group took place on June 24, 2016, in the premises of Decathlon Campus (Villeneuve D’Ascq, 59). I was seduced by the format and the relaxed atmosphere, and I have kept an excellent memory of it.

However, the regularity of the meetings did not meet the expectations of the project. The next event was scheduled more than a year later, on October 17, 2017, and it was the only one I could not attend. Guillaume and Pierre’s professional commitments did not allow them to keep up the pace, and the group remained dormant for more than three years.

I took the decision to take over the group in 2019, by offering Pierre to transfer me the administration rights of the group. My arrival at Dalibo as a PostgreSQL consultant gave me the opportunity to meet passionate experts, including Guillaume Lelarge and Stefan Fercot, who encouraged me to take over the project.

With a quality professional network on Twitter at the time, I was able to quickly mobilize a team of volunteers to organize the Meetup of January 28, 2020. I have been supported by Stefan Fercot (Dalibo), Stéphane Definin (Think) and Sébastien Freiss (SFEIR) to make this renewal a small success.

… Unfortunately, in March of the same year, the COVID-19 pandemic forced the group to suspend its projects and I did not have the courage to propose online content, much to the chagrin of the group’s members. For the record, no other Meetup group in France was spared by the health crisis and the restrictions in force.

Two additional years of pause were imposed on the group, before the health situation stabilized. In 2022, I contacted Lætitia Avrot, a very active member of the French community, to revive activities and find a dynamic and a rhythm for its members. The event of April 14, 2022 was a relief for me, seeing that I could still count on the participation of the community.

To this day, no more shadows have come to darken the picture, and the group has been able to resume the regular organization of meetings, from two to three per year. The community has expanded, and the organizing committee has been strengthened with the arrival of Yoann La Cancellera in 2023, who has allowed in particular to submit a request for recognition within the international community, on February 24, 2023.

How-to Meetup

To conclude this feedback, I wanted to share all the necessary steps to organize a Meetup, focusing on the crucial points of success and pitfalls to avoid.

Welcome : find the venue for the event

As the PG Day France changes location every year, the Meetup PG Lille group has chosen to vary the venues for each meeting. This allows you to discover new spaces and stimulate inter-community curiosity. Local companies are often delighted to be able to host technical events, and this helps to strengthen ties between local actors.

Searching for a venue is often the first step in organizing a Meetup. It is important to find a space that can accommodate between ten and forty people, with access to public transport or parking nearby. To do this, I have had the opportunity to use several strategies:

Chinest whispers: ask your professional contacts if they know of potential hosting locations
Social networks: post an announcement on Twitter or LinkedIn to solicit spontaneous proposals or leads to explore
Privatization of a space: contact coworking spaces or meeting rooms to get a rental quote
Partnering with other groups: propose a partnership with another Meetup group to share rental costs or to benefit from an already identified hosting location

Speakers: find two varied presentations

From the beginning, Pierre and Guillaume have chosen to offer two technical presentations to vary the topics and formats. This allows to reach a wider audience and to meet the expectations of the group’s members. This formula has been used for subsequent meetings, and has been a success every time.

Finding speakers is a time-consuming and patience-testing task. Often, a chance meeting or an informal discussion can lead to a presentation proposal. It is best to build a pool of resource persons who can be called upon when needed. It is not always easy, it is a long-term job, but it is worth it.

Spread the word, ask your colleagues, friends, and professional contacts if they are interested in sharing their experience or expertise, or if they know someone who might be interested.

Networking: extend the evening with a community highlight

From my point of view, this is the most important step in organizing a Meetup. (laughs!)

It’s the moment when group members can exchange, discuss, share, and meet. It’s a highlight of the evening that I particularly cherish, and that is intimately linked to the success of the event. Networking is what best promotes serendipity, chance encounters, sharing experiences, and creating lasting bonds. I discuss the group’s upcoming projects with members and the organizing committee, and I take notes on everyone’s expectations and listen to their suggestions.

Of course, it is important to provide drinks and snacks to extend the evening. It is necessary to provide a space to facilitate circulation and the natural creation of small discussion groups. The choice of a sponsor is mainly motivated by the coverage of these costs, with an invoice or an expense report as proof.

Until now, no incidents have been reported, and participants respect the few safety and conduct rules that are announced at the beginning of the evening.

Communication: reach the registration limit

This step is to be taken seriously a month before the event. It is about maximizing the visibility of the event to reach the maximum capacity set by the venue hosting the event. To do this, I have had the opportunity to use several methods:

Beeing as exhaustive as possible in the event description (date, location, schedule, program, sponsors, visuals, etc.)
Creating regular content on social networks to remind the date of the event and the registration modalities (LinkedIn majority, Twitter in the past)
Asking event participants to share the event on their social networks or their company networks
Informing the French community with the pgsql-fr-generale mailing list to reach a wider audience, or even to inspire other groups to organize Meetups (who knows?)

Visual identity (optional)

We worked with Yoann on the redesign of the Meetup group’s visual identity with a new logo and a new graphic charter in the last few months. It’s cosmetic, but it allows us to give a more professional image and a strong identity within the existing Lille communities. We hope to make a small place and a small notoriety in the local landscape.

We have delegated the creation of the event thumbnail twice, and I recommend working with a graphic designer or a designer to get a professional and neat result. Until one of the members around me is qualified enough, we do with the means at hand.

Last but not least, and it was our little pride of the year, we have given ourselves a Slonik logo, the PostgreSQL mascot. The creation process relied heavily on the new AI generation tools, and we were blown away by the quality of the result. A big thank you to Isaac for his invaluable help in the retouching and adjustments of the logo.

Acknowledgements

During this conference, I thanked a lot of people. In particular, the members of the PostgreSQL France association, for their trust they have given us, Matthieu Cornillon and myself, for accepting our application file for the choice of the city of Lille this year 2024.

One of the slides of my presentation was dedicated to all the companies that hosted our meetings, and who, without knowing it, allowed me to adjust the format and content of our Meetups. A thank you to them for their sympathy and warm welcome. It was also an opportunity to highlight the speakers who agreed to share their experience and expertise during the PostgreSQL Lille Meetups. A big thank you to them for living the adventure with me!

After the conference, I felt that feedback had a positive impact on the audience, with a few people coming to me to congratulate me on the work done and inform me of their intention to organize a Meetup in their own city. It was a nice reward for me, and it made me want to continue my commitment and set an example for the promotion of PostgreSQL in France.

On the road to freedom with db_migrator

Florent Jardin — Fri, 28 Jul 2023 00:00:00 +0000

Over the past months, I have spent several weeks contributing to the db_migrator extension. Written solely in PL/pgSQL, it enables the migration of schemas and data from a database system to PostgreSQL using the external data I had presented few years ago.

In this post, I present the functionality of the tool, its philosophy, and the reason I found for its existence, even though it joins the ecosystem of well-established open-source projects in the migration landscape. How does it compare to Ora2Pg or pgloader in terms of value and capabilities?

db_migrator enters the arena

My interest in this project dates back to last December when a colleague from Dalibo left us a similar tool, which allowed copying data from Oracle or Sybase instances using Foreign Data Wrappers (FDW) technology. Although this tool remained in alpha, many good ideas were experimented with internally.

The promise of FDWs lies in adhering to the SQL/MED standard, allowing a PostgreSQL instance to interface with another storage system and manipulate its data through external tables using simple SQL queries. Therefore, provided that a community has developed the wrapper, it becomes possible to query a remote catalog, replicate the structure of tables, their relationships, and constraints, and retrieve data into PostgreSQL.

And db_migrator enters the arena.

Made public in November 2019 by Laurenz Albe, well-known for his active contributions to PostgreSQL for decades and also for developing oracle_fdw, the extension presents itself as a generic tool where one must use plugins for FDW support. It is easy to create new plugins, as I discovered with the mysql_migrator plugin, written in just a few days, thanks to the comprehensive documentation of the API for plugins.

After installing the extensions with make install and the appropriate FDW for the system, it is necessary to create the objects in the database that will hold the future schemas and their data.

CREATE EXTENSION mysql_fdw;
CREATE EXTENSION mysql_migrator CASCADE;

CREATE SERVER mysql FOREIGN DATA WRAPPER mysql_fdw
 OPTIONS (host 'mysql_db', fetch_size '1000');
CREATE USER MAPPING FOR PUBLIC SERVER mysql
 OPTIONS (username 'root', password 'password');

The migration process can be performed in a single command for the simplest cases (no stored procedures or exotic column types) using the db_migrate() method. Otherwise, for more complex scenarios requiring adjustments such as changing column types or removing a table in the target schema, the migration may involve multiple steps.

During the development of the mysql_migration extension, I started with the sample database Sakila provided by MySQL to have comprehensive complexity. The first step involves creating two internal schemas, one with external tables provided by the plugin and the other with catalog tables that can be edited before the extension continues the migration.

SELECT db_migrate_prepare(
 plugin => 'mysql_migrator',
 server => 'mysql',
 only_schemas => '{sakila}'
);

This part can be relatively lengthy, as it involves retrieving the data model, which I refer to as the catalog, in the form of several tables that describe the structure of tables, column names, and associated constraints. The extension also imports the sources of all stored procedures, functions, and views but does not perform their conversion to PL/pgSQL (you cannot imagine the amount of work involved).

For the migration of the Sakila database, several modifications to the catalog are necessary. Like the rest of this extension, all the preparation is done in SQL, making it easy to automate with a single script serving as configuration.

/* exclude bytea columns from migration */
DELETE FROM pgsql_stage.columns WHERE type_name = 'bytea';

/* quote character expression */
UPDATE pgsql_stage.columns
 SET default_value = quote_literal(default_value)
 WHERE NOT regexp_like(default_value, '^\-?[0-9]+$')
 AND default_value <> 'CURRENT_TIMESTAMP';

/* disable view migration */
UPDATE pgsql_stage.views SET migrate = false;

Of course, we could go further, such as reinjecting the definition of rewritten views into the pgsql_stage.views table or enabling the migration of procedures by changing the migrate column of the pgsql_stage.functions table. However, let’s proceed with the next step.

SELECT db_migrate_mkforeign(
 plugin => 'mysql_migrator',
 server => 'mysql'
);

SELECT db_migrate_tables(
 plugin => 'mysql_migrator'
);

The first method, db_migrate_mkforeign(), is responsible for creating schemas and sequences, followed by foreign tables with columns based on the previous adjustments. Next comes the most crucial step, where we execute the function db_migrate_tables(): blank tables are created with their partitions if necessary, and for each of them, the data copying begins using the INSERT INTO SELECT * statement.

Other objects, such as indexes or constraints, have their own methods. It is necessary to create the functions before these objects if you encounter functional indexes or other similar cases.

SELECT db_migrate_functions(plugin => 'mysql_migrator');
SELECT db_migrate_triggers(plugin => 'mysql_migrator');
SELECT db_migrate_views(plugin => 'mysql_migrator');
SELECT db_migrate_indexes(plugin => 'mysql_migrator');
SELECT db_migrate_constraints(plugin => 'mysql_migrator');

It is possible that this mechanism may change in the future, especially if I manage to realize this issue, which would allow breaking down the db_migrate_*() methods into smaller steps.

The end of the migration process involves deleting the temporary schemas that contained the catalog tables.

SELECT db_migrate_finish();

One more migration tool

As I mentioned in the introduction, it is quite surprising to see a new migration tool emerge in 2023 (the version 1.0.0 was released in January with my patch on partitioning). In the open-source landscape, we can mention Ora2Pg, which released its version 24.0 in July with SQL Server support, and pgloader, which has an excellent reputation.

A vast number of projects are listed on the community wiki. Some are specialized for a single system, while others support migration for multiple systems. The majority of these projects are either proprietary or lack recent contributions. Many of them are black boxes, and their documentation may appear cryptic or almost non-existent.

The ecosystem is rich, and I do not claim to know all of its aspects, but I have had an intuition that I have been forming over the past few years. The global economy is in a state of turmoil. Some companies are doing well, while others are making budget cuts. The transition to a free and non-commercially licensed system like PostgreSQL remains relevant, perhaps even more urgent today compared to the past decade.

And yet, with my DBA perspective, I am not fully satisfied with the existing tools. I wish for a new alternative, something universal and accessible to everyone. If I turn to db_migrator today, it would be for the following main advantages:

A low-level implementation close to the instance: using PL/pgSQL as the exclusive language. This would not have been possible without the prolific development of Foreign Data Wrappers for a wide range of systems;
A high level of configuration flexibility: adjustments are made with UPDATE or DELETE queries on the catalog. Once one is familiar with the model of the catalog, it becomes easy to change behavior without consulting technical documentation on the available options;
Freedom in orchestration: currently, executions are triggered sequentially for indexes and constraints, but the tool’s architecture could allow external tools to consume the extension’s results and trigger operations in parallel;
Plugins have the freedom to enrich migration: if an operation is not generic, it is entirely possible to provide an additional method through the plugin. For example, the incremental copy (and its replication functions) in the ora_migrator plugin or the conversion of auto-increments to identity columns with the mysql_migrator plugin.

The road to freedom still seems long to achieve half of what Ora2Pg already offers, especially when it comes to automatic conversion, which is not on the agenda at all. But with small, regular, and thoughtful advancements, who knows?