import ibis
import ibis.selectors as s
from ibis import _
# ibis.options.interactive = True # enforce eager execution of queriesDuckDB + Ibis (Python)
Use a dplyr-esque Python frontend
Load libraries
Connect and register
## Instantiate an in-memory DuckDB connection from Ibis
con = ibis.duckdb.connect()
## Register our parquet dataset as a table called "nyc" in our connection
nyc = con.read_parquet("nyc-taxi/**/*.parquet")Aside: Remember that you can create a persistent, disk-backed database by giving it an appropriate name/path. This also enables out-of-core computation for bigger than RAM data.
# con = ibis.duckdb.connect("nyc.dbb")
# nyc = con.read_parquet("nyc-taxi/**/*.parquet")
# etc.Reference the table from Python. We’ll call this reference object nyc too for consistency, but you could call it whatever you want (e.g., you could call it nyc_ibis to avoid potential ambiguity with the “nyc” table in our actual DuckDB connection). Printing the object to screen will give us a lazy preview.
nycDatabaseTable: ibis_read_parquet_ubbotra4snbqjhfys5d3pmq44e vendor_name string pickup_datetime timestamp(6) dropoff_datetime timestamp(6) passenger_count int64 trip_distance float64 pickup_longitude float64 pickup_latitude float64 rate_code string store_and_fwd string dropoff_longitude float64 dropoff_latitude float64 payment_type string fare_amount float64 extra float64 mta_tax float64 tip_amount float64 tolls_amount float64 total_amount float64 improvement_surcharge float64 congestion_surcharge float64 pickup_location_id int64 dropoff_location_id int64 month int64 year int64
First example
q1 = (
nyc
.group_by(["passenger_count"])
.agg(mean_tip = _.tip_amount.mean())
)To see the underlying SQL translation, use ibis.to_sql()
ibis.to_sql(q1)SELECT
"t0"."passenger_count",
AVG("t0"."tip_amount") AS "mean_tip"
FROM "ibis_read_parquet_ubbotra4snbqjhfys5d3pmq44e" AS "t0"
GROUP BY
1To actually execute the query and bring the result into Python, we can use the execute() method. By default this will coerce to a pandas DataFrame.
dat1 = q1.execute()
dat1| passenger_count | mean_tip | |
|---|---|---|
| 0 | 249 | 0.000000 |
| 1 | 0 | 0.862099 |
| 2 | 254 | 0.000000 |
| 3 | 8 | 0.350769 |
| 4 | 66 | 1.500000 |
| 5 | 2 | 1.081580 |
| 6 | 208 | 0.000000 |
| 7 | 10 | 0.000000 |
| 8 | 1 | 1.151011 |
| 9 | 247 | 2.300000 |
| 10 | 3 | 0.962949 |
| 11 | 6 | 1.128365 |
| 12 | 5 | 1.102732 |
| 13 | 65 | 0.000000 |
| 14 | 9 | 0.806800 |
| 15 | 177 | 1.000000 |
| 16 | 7 | 0.544118 |
| 17 | 4 | 0.844519 |
NoteIbis conversion to polars
The q1.execute() method above is equivalent calling q1.to_pandas(). A q1.to_polars() equivalent has been added to the dev version of Ibis, but is not available with the latest offical release (8.0.0 at the time of writing).
Digression: Interactive Ibis use and eager execution
At the very top of this document, I commented out the ibis.options.interactive option as part of my Ibis configuration. This was because I wanted to demonstrate the default deferred (i.e., lazy) behaviour of Ibis, which is just the same as d(b)plyr in R. If you are building data wrangling pipelines, or writing scripts with potentially complex queries, you probably want to preserve this deferred behaviour and avoid eager execution.
However, there are times when you may want to default into eager execution. For example, if your dataset is of manageable size, or you are trying to iterate through different query operations… Or, you might just want to enable it so that you automatically get a nice print return object for your workshop materials. I’ll adopt the latter view, so that I can quickly demonstrate some Ibis syntax and results for the rest of this document.
ibis.options.interactive = TrueOkay, let’s speed through some of the same basic queries that we’ve already seen in the DuckDB SQL and R (dplyr) pages. I won’t bother to explain them in depth. Just consider them for demonstration purposes.
Aggregation
(
nyc
.group_by(["passenger_count", "trip_distance"])
.aggregate(
mean_tip = _.tip_amount.mean(),
mean_fare = _.fare_amount.mean()
)
)┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ passenger_count ┃ trip_distance ┃ mean_tip ┃ mean_fare ┃ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩ │ int64 │ float64 │ float64 │ float64 │ ├─────────────────┼───────────────┼──────────┼───────────┤ │ 1 │ 5.3 │ 1.812836 │ 17.073446 │ │ 2 │ 3.2 │ 1.144488 │ 12.271308 │ │ 0 │ 3.2 │ 1.143411 │ 11.369103 │ │ 1 │ 8.9 │ 3.222421 │ 24.642239 │ │ 1 │ 17.4 │ 5.635788 │ 47.341387 │ │ 4 │ 1.2 │ 0.397633 │ 6.819041 │ │ 1 │ 2.4 │ 1.069486 │ 10.005681 │ │ 3 │ 16.9 │ 3.056976 │ 47.266028 │ │ 3 │ 17.8 │ 2.755452 │ 48.073206 │ │ 1 │ 3.1 │ 1.278962 │ 11.872071 │ │ … │ … │ … │ … │ └─────────────────┴───────────────┴──────────┴───────────┘
Note that, even though we have enabled the interactive print mode, we still get lazy evalation if we assign a chain of query steps to an object (here: q3)…
q3 = (
nyc
.group_by(["passenger_count", "trip_distance"])
.agg(
mean_tip = _.tip_amount.mean(),
mean_fare = _.fare_amount.mean()
)
)… but printing the query to screen enforces computation.
q3┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ passenger_count ┃ trip_distance ┃ mean_tip ┃ mean_fare ┃ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩ │ int64 │ float64 │ float64 │ float64 │ ├─────────────────┼───────────────┼──────────┼───────────┤ │ 1 │ 11.10 │ 3.839358 │ 29.974973 │ │ 2 │ 2.60 │ 0.973256 │ 10.619201 │ │ 1 │ 3.50 │ 1.384175 │ 12.883433 │ │ 1 │ 2.30 │ 1.040463 │ 9.740110 │ │ 3 │ 1.86 │ 0.777333 │ 8.460294 │ │ 1 │ 0.90 │ 0.578975 │ 5.687596 │ │ 2 │ 6.80 │ 1.977422 │ 20.725585 │ │ 2 │ 1.70 │ 0.727560 │ 8.173302 │ │ 1 │ 0.81 │ 0.513297 │ 5.402281 │ │ 1 │ 0.77 │ 0.499988 │ 5.271770 │ │ … │ … │ … │ … │ └─────────────────┴───────────────┴──────────┴───────────┘
Pivot (reshape)
# now chain on pivoting (and enforce computation via printing)
(
q3
.pivot_longer(["mean_tip", "mean_fare"])
)┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ passenger_count ┃ trip_distance ┃ name ┃ value ┃ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩ │ int64 │ float64 │ string │ float64 │ ├─────────────────┼───────────────┼───────────┼───────────┤ │ 1 │ 11.10 │ mean_tip │ 3.839358 │ │ 1 │ 11.10 │ mean_fare │ 29.974973 │ │ 2 │ 2.60 │ mean_tip │ 0.973256 │ │ 2 │ 2.60 │ mean_fare │ 10.619201 │ │ 1 │ 3.50 │ mean_tip │ 1.384175 │ │ 1 │ 3.50 │ mean_fare │ 12.883433 │ │ 1 │ 2.30 │ mean_tip │ 1.040463 │ │ 1 │ 2.30 │ mean_fare │ 9.740110 │ │ 3 │ 1.86 │ mean_tip │ 0.777333 │ │ 3 │ 1.86 │ mean_fare │ 8.460294 │ │ … │ … │ … │ … │ └─────────────────┴───────────────┴───────────┴───────────┘
Joins (merges)
(As we did in the dplyr code, we’ll break this contrived join example into two steps)
mean_tips = nyc.group_by("month").agg(mean_tip = _.tip_amount.mean())
mean_fares = nyc.group_by("month").agg(mean_fare = _.fare_amount.mean())(
mean_tips
.left_join(mean_fares, "month")
)┏━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ month ┃ mean_tip ┃ month_right ┃ mean_fare ┃ ┡━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ int64 │ float64 │ int64 │ float64 │ ├───────┼──────────┼─────────────┼───────────┤ │ 8 │ 1.079521 │ 8 │ 10.492650 │ │ 6 │ 1.091082 │ 6 │ 10.548651 │ │ 9 │ 1.254601 │ 9 │ 12.391198 │ │ 5 │ 1.078014 │ 5 │ 10.585157 │ │ 7 │ 1.059312 │ 7 │ 10.379943 │ │ 3 │ 1.056353 │ 3 │ 10.223107 │ │ 2 │ 1.036874 │ 2 │ 9.942640 │ │ 4 │ 1.043167 │ 4 │ 10.335490 │ │ 12 │ 1.237651 │ 12 │ 12.313953 │ │ 11 │ 1.250903 │ 11 │ 12.270138 │ │ … │ … │ … │ … │ └───────┴──────────┴─────────────┴───────────┘
Disconnect
con.disconnect()