Stuff about Things

Data Loading Performance of Postgres and TimescaleDB

Fabien — Fri, 13 Sep 2019 00:00:00 +0000

Data Loading Performance of Postgres and TimescaleDB

Postgres is the leading feature-full independentopen-source relational database, steadily increasing its popularity over thepast 5 years.TimescaleDB is a clever extension to Postgreswhich implements time-series related features, including under the hoodautomatic partioning, and more.

Because he knows how I like investigate Postgres (among other things)performance,Simon Riggs(2ndQuadrant) prompted me to look at theperformance of loading a lot of data into Postgres and TimescaleDB, so as tounderstand somehow the degraded performance reported in theirTimescaleDB vs Postgrescomparison.Simon provided support, including provisioning 2 AWS VMs for a few days each.

Choosing Postgres Bloom Index Parameters

Fabien — Sun, 11 Dec 2016 00:00:00 +0000

Choosing Postgres Bloom Index Parameters

Postgres Bloom is anextension contributed by Teodor Sigaev, Alexander Korotkov andOleg Bartunov which provides a new index type for integer and text columns.There is some coverage on how to use it and how it works, which is good becausedocumentation is scarse.

Postgres FILLFACTOR for UPDATE

Fabien — Sat, 23 Aug 2014 00:00:00 +0000

Postgres `FILLFACTOR` for `UPDATE`

This post discusses the performance impact of Postgres FILLFACTOR tablestorage parameter on an UPDATE OLTP load.Note that this FILLFACTOR is indeed the table storage parameter, althoughthere is also an eponymous parameter for indexes.

Postgres page size for SSD (2)

Fabien — Sun, 17 Aug 2014 00:00:00 +0000

Postgres page size for SSD (2)

I have recently posted performance figures withvarying page size using pgbench on SSD, which show a +10% improvement withsmaller 4 kB page size over the default 8 kB page size.

Josh Berkus pointed out that pgbench test uses rather small 100-bytes rows, andthat changing the tuple size might induce a different conclusion.To assess this point, I ran some tests with different row sizes:1 kB, 3 kB, 7 kB and 15 kB.

Postgres page size for SSD

Fabien — Fri, 08 Aug 2014 00:00:00 +0000

Postgres page size for SSD

In a previous post, I have outlinedthe time required by a Postgres database to warm-up from a HDD on a read-onlyload for a database that fits in memory.

In this post, I want to look at write performance on SSD, focusing on the impactof Postgres page size (blocksize), and on checking whether the current 8 kBdefault is relevant.

DataFiller 2.0.0 is out!

Fabien — Sun, 23 Mar 2014 00:00:00 +0000

DataFiller 2.0.0 is out!

DataFiller processes aPostgres database schema file augmented with directives in comments, andgenerates pseudo-random data matching this schema, taking into accountconstraints such as types, but also primary key, unique, foreign keys, not null…

Version 2.0.0 introduces the following new features:

DataFiller Tutorial

Fabien — Sun, 01 Dec 2013 00:00:00 +0000

DataFiller Tutorial

This tutorial introduces how to use DataFiller to fill a Postgres database, sayfor testing functionalities and performances.

DataFiller 1.1.3 is out!

Fabien — Fri, 29 Nov 2013 00:00:00 +0000

DataFiller 1.1.3 is out!

I have just released version 1.1.3 of DataFiller.

The Python script processes a Postgres database schema file augmented withdirectives in comments, and generates random data matching this schema, takinginto account constraints such as types, but also primary key, unique, foreignkeys, not null…

Postgres Archeology

Fabien — Sun, 01 Sep 2013 00:00:00 +0000

Postgres Archeology

In a previous post I have ported a Turing Machine (TM) in SQL down to Postgres 7.3.

I report here my fruitless attempts at running previous versions of Postgres ona Debian or Ubuntu Linux.They would not configure and/or compile, at least with the minimal effort I wasready to undergo: writing portable-to-the-future system-dependent C code doesnot work.Basically, the code from 15 years ago is lost unless you run it on a systemfrom 15 years ago, which may require a hardware from 15 years ago as well, ormaybe a VM.

Turing Machine in SQL (5/5)

Fabien — Thu, 29 Aug 2013 00:00:00 +0000

Turing Machine in SQL (5/5)

This is the last of five posts on this subject.

In a previous post I have shown how to build a Turing Machine(TM) in SQL with Postgres using a recursive SQL function.I claimed that this would work with versions of Postgres older than 8.4.In this post, I actually investigate this claim, porting the script down toversion 7.3 on a Debian Linux.Testing the relatively simple SQL script with older versions reminds us of allthe goodies that have been added over the years.

Turing Machine in SQL (4/5)

Fabien — Mon, 26 Aug 2013 00:00:00 +0000

Turing Machine in SQL (4/5)

This is the fourth of five posts on this subject.

In previous posts,I have presented different ways of implementing a Turing Machine (TM) in SQLwith Postgres.All three techniques rely on WITH RECURSIVE to iterate till the TM stops, soas to provide some kind of while construct.

In this post, I get rid of this construct, so that the solution does not requirePostgres 8.4 or later.Obviously there is a trick: I will use a recursive SQL function with sideeffects on a TABLE to execute the TM.

Turing Machine in SQL (3/5)

Fabien — Fri, 23 Aug 2013 00:00:00 +0000

Turing Machine in SQL (3/5)

This is the third of five posts on this subject.

In previous posts, I have presented howto implement a Turing Machine (TM) with the tape stored as an ARRAY or in aseparate TABLE accessed through SQL functions.In this post the solution is more cleanly relational, with the tape contentsstored in a column of the recursive query, very likeAndrew Gierth’s CTS implementation.

Turing Machine in SQL (2/5)

Fabien — Tue, 20 Aug 2013 00:00:00 +0000

Turing Machine in SQL (2/5)

This is the second of five posts on this subject.

In a previous post I presented how to implement a TuringMachine (TM) with an ARRAY to store the tape contents.This solution is not reallyrelational, so in this postI’ll show how to build a TM with SQL-only functions.

Turing Machine in SQL (1/5)

Fabien — Sat, 17 Aug 2013 00:00:00 +0000

Turing Machine in SQL (1/5)

This is the first of five posts on this subject.

Postgres implementation of SQL isTuring Complete, as show byAndrew Gierth on Postgres wiki,where aCyclic Tag System (CTS), whichis proven Turing-complete, isimplemented using WITH RECURSIVE, a WINDOW function, CASE conditionalexpressions and an OUTER JOIN.Although the proof is sound, there is a long path from a CTS to an actualTuring Machine (TM).

The first question I want to answer is: how to build a Turing Machine with SQLonly?

The second question I would like to investigate is: What SQL features areactually required to build a relational Turing Machine.For instance, can it be done without WITH RECURSIVE, WINDOW functions andOUTER JOIN? (Teaser for later posts: yes!)

Postgres Warm-up

Fabien — Wed, 14 Aug 2013 00:00:00 +0000

Postgres Warm-up

A database can take some time to reach good performances, the time necessary toOS and database caches to load the necessary data from the hard disk drive.Here is an example with Postgres, which outlines the long delay incurred(18 minutes), presents a simple model to explain this behavior, and shows how toreduce it significantly (to a few minutes).