How Clones Work

Dubhub leverages a shared library hook that sits on top of your operating system to create and clone snapshots - without modifying the database binary, changing the OS, or installing special drivers.

For clarity, we’ll refer only to PostgreSQL in this article, though our technology also supports MySQL, MariaDB, and MongoDB - with more on the way!

This article covers:

1. How Applications Request Data from the OS
2. How shared library hooking works
3. How Dubhub creates writable clones from snapshots
4. Dubhub Docker image layers

How Applications Request Data from the OS

Typically, applications don’t interact directly with the operating system (OS). Instead, they access the file system through standard shared libraries. These libraries aren’t bundled into the application binary; instead, the application tells the OS which shared libraries it depends on. These libraries are referred to as dependencies.

When an application is started, and it uses a shared library, the operating system recognizes the dependency and loads it from a set of known, installed libraries. One common example is libc, provided by the OS, which translates application requests into system calls (syscalls). These syscalls are then handled by various OS modules, which return the result to the application

A good example is when an application calls the pread() function, commonly used to read data from a file. Below is a simplified call flow for a database query that retrieves data from disk:

1. A user connects to the database and issues a query to a users table: SELECT * FROM users.
2. The database engine calls pread() to read data from the file backing the users table.
3. The pread() function, implemented in libc, translates the request into a syscall.
4. The syscall is handled by the operating system’s components (e.g., file system and drivers).
5. The result makes its way back up the call chain, through libc, until it reaches the call in the database program.

How shared library hooking works

Like any other dynamic shared library, libc is not part of the database binary - it’s provided by the operating system. When the database binary resides on disk, it includes references (or "stubs") to the shared libraries it depends on. The stubs essentially function like a translating system, explaining to the operating system what libraries are needed, and which functions they implement.

When the database program starts:

1. It calls functions from shared libraries, such as pread() from libc.so.6.
2. This triggers a process called dynamic loading.
3. During dynamic loading, an appropriate shared library is identified and loaded into memory if necessary, linking the stubs to the loaded shared library.

The dynamic loading process enables the database to load the libc library. It also enables users to interfere with its process such that another library, separate from the one specified by the binary, is loaded instead; and shared libraries can load other shared libraries, including libc. In this way, a replacement for libc can be specified. This is known as libc hooking.

Dubhub's special sauce is a libc hook that creates the illusion of working on a writable copy of a snapshot, similar to a mount in copy-on-write (CoW) file systems, but without the related hassle of installing drivers, CoW filesystems, or otherwise modifying or hooking the operating system itself.

How Dubhub creates writable clones from snapshots

A Dub maintains a sanitized database replica of a source database. Dubhub periodically creates a snapshot of the sanitized replica database. The data of a snapshot is broken down into files called chunks. When running locally, the local database service (LDS) runs PostgreSQL (with the Dubhub libc hook) using a combination of read-only and read-write chunks.

LDS maintains two types of chunks:

• read-only chunks - a set of read-only data files. Dubhub delivers snapshots as a set of chunks that contain data from a database snapshot, together with a snapshot metadata file created by the Dub. The snapshot metadata contains file system metadata (directories, file names, permissions, etc), together with file structure metadata relating files to their respective chunks. Chunks can be shared between snapshots, such that downloading new snapshots only requires retrieving the latest snapshot metadata file and the changed chunks.
• read-write chunks - a read-write directory of files created and maintained per clone.

The Dubhub libc hook (managed by LDS) seamlessly routes database commands like pread() and pwrite() to read-only and read-write chunks in real time.

Following is a (simplified) example of LDS and the Dubhub libc hook in action when launching a clone of a snapshot, followed by a change of data in a table, and the subsequent querying of the changed data.

1. LDS creates a directory for the clone a6ee72
2. LDS starts a PostgreSQL instance with the Dubhub libc hook, configured to read from snapshot 20210825 and maintain changes in clone directory a6ee72
3. The user connects to the database and issues the SQL command: UPDATE users SET name = "joe" WHERE id = 5
4. PostgreSQL attempts to modify the file containing users table data by issuing a write() command.
5. The libc hook intercepts the write() command and:
   • recognizes that it is attempting to modify data in a snapshot file
   • reads this data from the snapshot file
   • applies the modification
   • saves the changed data to a file in the clone's directory
   • records the change to the file
6. The user runs an SQL command SELECT * FROM users WHERE id = 5. This command reads data in a users table that exists in the snapshot and was modified in the clone.
7. Finally, an SQL query retrieves the modified data. The user issues the new query to PostgreSQL, which attempts to read from the file containing the users table data by issuing a read() command.
8. The libc hook intercepts the read() command and:
   • identifies that the read() retrieves data from a modified area of a chunk
   • reads from the modified chunk

Dubhub Docker image layers

The Dubhub base Docker image contains an installation of PostgreSQL, LDS, and a libc hook. On top of this image, Dubhub adds one layer per snapshot, each containing chunks and a snapshot metadata file.

The first layer on the base image contains a full snapshot, meaning it contains a copy of the whole snapshot, which is relatively large in size. Starting from the second layer (representing subsequent snapshots), only data in chunks that have changed since the previous snapshot is included. This typically results in layers ~2-7% the size of a full snapshot.

Above is a visualization of the layers of the second daily snapshot of a database whose initial snapshot is 12GB. The base layer contains the PostgreSQL and Dubhub binaries. The two daily snapshots are each a fraction of the size of the base snapshot.

info

Due to a limitation on the number of layers in a Docker image, Dubhub creates a new base snapshot for every 30 layers