Pagination

Most Variational API endpoints that can return multiple results limit the number of objects returned by a single call. This means that a pagination mechanism must be used to retrieve more objects.

Endpoints using pagination can be identified by the presence of the top-level pagination key in the JSON data they return. In the Endpoint Reference their Python SDK return value is marked as ApiPage[T].

Different endpoints return different number of items on a single page. The Endpoint Reference specifies this number for each endpoint.

Example

When requesting the first page of trades, the response will contain query string parameters required to request the next page.

Example first request:

GET https://api.testnet.variational.io/v1/portfolio/trades?pool=daf1b8ef-bac8-47a0-8d41-f18c9aed8442

Example response:

{
    "result": [{ ... }, { ... }, ...],
    "pagination": {
        "next_page": {
            "limit": "1000",
            "offset": "1000"
        }
    }
}

This means that the next page can be retrieved using the URL:

GET https://api.testnet.variational.io/v1/portfolio/trades?pool=daf1b8ef-bac8-47a0-8d41-f18c9aed8442&limit=1000&offset=1000

The last page of the results will still contain the pagination key but next_page inside of it will be null:

{
    "result": [{ ... }, { ... }, ...],
    "pagination": {
        "next_page": null
    }
}

Python SDK

For flexibility, the Python SDK Client methods don't perform the pagination automatically:

from variational import Client, TESTNET

client = Client(API_KEY, API_SECRET, base_url=TESTNET)
trades_resp = client.get_portfolio_trades(pool="daf1b8ef-bac8-47a0-8d41-f18c9aed8442")
trades = trades_resp.result  # only the first page of trades

Manual Pagination

If necessary, the next page can be retrieved afterward:

if trades_resp.pagination.next_page:
    next_resp = client.get_portfolio_trades(
        pool="daf1b8ef-bac8-47a0-8d41-f18c9aed8442",
        page=trades_resp.pagination.next_page
    )
    next_trades = next_resp.result

Using the Helper

To avoid boilerplate, the SDK provides a helper generator function paginate() that yields the requested objects making API calls lazily while it's being consumed.

Signature:

def paginate(method: Callable[..., ApiPage[T]],
             *args, page=None, **kwargs) -> Generator[T, None, None]

This code fetches all trades from the previous example into a list using the helper function:

from variational import paginate

# Note that we don't call `client.get_portfolio_trades` here,
# but rather pass a reference to it to `paginate()`.
# All parameters meant for `client.get_portfolio_trades`
# should be passed to `paginate()`.
# `list()` is meant to consume all results from the generator
# at once and eagerly request all pages.
all_trades = list(paginate(client.get_portfolio_trades,
                           pool="daf1b8ef-bac8-47a0-8d41-f18c9aed8442"))

An example making use of lazy pagination:

pool_id = "daf1b8ef-bac8-47a0-8d41-f18c9aed8442"
for trade in paginate(client.get_portfolio_trades, pool=pool_id):
    if is_this_the_last_trade_we_need(trade):
        break  # no more API calls requesting subsequent pages will happen after this

Logging

If you have a large amount of activity in your account, it might take multiple requests and a significant amount of time to fetch all resulting objects using pagination. Running multiple pagination requests back to back makes it likely to run into Rate Limits. The SDK Client retries them automatically by default but the entire operation will become slowed down.

It might be useful to enable logging in your application to get an indication of progress.

A simple logging configuration that enables debug-level logging for all facilities:

import logging

# run this before initializing variational.Client
logging.basicConfig(
    format="[%(levelname)s] %(asctime)s %(name)s %(message)s",
    level=logging.DEBUG
)

Advanced configuration enabling only relevant loggers:

import logging, logging.config

logging.config.dictConfig({
    "version": 1,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "level": "DEBUG",
            "formatter": "default",
        }
    },
    "formatters": {
        "default": {
            "format": logging.BASIC_FORMAT,
            "datefmt": "%Y-%m-%d %H:%M:%S",
        }
    },
    "loggers": {
        "variational": {  # logs specific to Variational
            "handlers": ["console"],
            "level": "DEBUG",
            "propagate": True,
        },
        "urllib3.connectionpool": {  # logs of all HTTP requests being made
            "handlers": ["console"],
            "level": "DEBUG",
        },
    }
})