I was in need of importing a big SQLite database inside of DuckDB and I found the documentation of the sqlite extension of duckdb (Archived) but I couldn't find any reference to how to import an SQLite database.
I had a look at the duckdb folder structure of an export command, and I found that duckdb just executes the sql statements found inside schema.sql
and load.sql
, so that you can just create the schema and the instructions to load the dataset using the sqlite extension, and I wrote a simple script that does it:
#!/bin/bash
if [ "$1" == "" ]; then
echo "Convert an SQLite database with homogeneous types to DuckDB"
echo "Usage: $0 database.sqlite3"
exit 1
fi
SQLITEDB=$1
DBNAME=${SQLITEDB%.*}
DUCKDB="${DBNAME}.duckdb"
sqlite3 "${SQLITEDB}" .schema > schema.sql
TABLES=$(IFS=$'\n'; cat schema.sql | grep -oP "^CREATE TABLE (IF NOT EXISTS )?\K.*?(?=\()")
IFS=$'\n'
echo "INSTALL sqlite;" > load.sql
echo "LOAD sqlite;" >> load.sql
for table in $TABLES; do
echo "INSERT INTO $table SELECT * FROM sqlite_scan(\"$SQLITEDB\", $table);"
done >> load.sql
duckdb "${DUCKDB}" "import database './'"
The script simply dumps the SQLite schema to schema.sql
, then uses grep to build an insert statement from the SQLite database for each table found in the schema, and later uses the import command of duckdb to import the current folder.
For me it has worked flawlessly, and much faster than exporting the SQLite database to a full sql
file and reimporting it into duckdb.
I hope the script can be useful to you too, but don't forget that it works only when SQLite tables have homogeneous datatypes, as duckdb is strict with types.