> ## Documentation Index
> Fetch the complete documentation index at: https://docs.dakota.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Bulk Data - Pagination

> Retrieve large datasets in pages using offset and max_num

## Overview

Use **pagination** when you need to retrieve large datasets in smaller batches. Dakota supports this through `max_num` (page size / limit) and `offset` (starting position) inside the `filters` object.

| Parameter  | Description                                                                               |
| :--------- | :---------------------------------------------------------------------------------------- |
| `max_num`  | Number of records to return per request (limit / page size). **Recommended range: 20–50** |
| `offset`   | Number of records to skip before returning results (defaults to `0` if omitted)           |
| `order_by` | Sort order for stable paging (recommended for bulk pulls)                                 |

### How offset and max\_num work

| Request | `offset` | `max_num` | Returns         |
| :------ | :------- | :-------- | :-------------- |
| Page 1  | `0`      | `50`      | Records 1–50    |
| Page 2  | `50`     | `50`      | Records 51–100  |
| Page 3  | `100`    | `50`      | Records 101–150 |

**Formula:** `offset = (page_number - 1) * max_num`

## Response shape

List responses return an envelope:

```json theme={null}
{
  "records": [ /* page of records */ ],
  "next_offset": 50
}
```

| Field         | Meaning                                                                    |
| :------------ | :------------------------------------------------------------------------- |
| `records`     | Array of records for the current page                                      |
| `next_offset` | Offset to use for the next request. `-1` means there are **no more pages** |

`count_only` responses return:

```json theme={null}
{
  "record_count": 160167
}
```

## How to Use

1. Optionally call with `count_only: true` to get `record_count`.
2. Request page 1 with `offset: 0` and your chosen `max_num`.
3. For the next page, set `offset` to the previous response’s `next_offset` (or add `max_num` to the current offset).
4. **Stop when `next_offset` is `-1`** (last page, empty result, or past end). You can also stop when `records.length < max_num`.

## Empty and last-page behavior

| Scenario                     | `records`                  | `next_offset`                                   |
| :--------------------------- | :------------------------- | :---------------------------------------------- |
| Full page                    | `max_num` items            | next page offset (for example `10`, `20`, `50`) |
| Last page (partial)          | fewer than `max_num` items | `-1`                                            |
| No matches (empty filter)    | `[]`                       | `-1`                                            |
| Offset at/beyond total count | `[]`                       | `-1`                                            |

## Account Examples

### First Page of Accounts

```json theme={null}
{
  "module": "account",
  "filters": {
    "fields": ["sfid", "name", "type", "aum__c", "lastmodifieddate"],
    "order_by": "lastmodifieddate:DESC",
    "max_num": 50,
    "offset": 0
  }
}
```

### Next Page of Accounts

```json theme={null}
{
  "module": "account",
  "filters": {
    "fields": ["sfid", "name", "type", "aum__c", "lastmodifieddate"],
    "order_by": "lastmodifieddate:DESC",
    "max_num": 50,
    "offset": 50
  }
}
```

### Paginate Accounts with a Filter

```json theme={null}
{
  "module": "account",
  "filters": {
    "fields": ["sfid", "name", "type", "aum__c", "lastmodifieddate"],
    "order_by": "name:DESC",
    "max_num": 50,
    "offset": 0,
    "filter": [
      {
        "aum__c": {
          "$between": [100, 500000]
        }
      }
    ]
  }
}
```

### Count Accounts Before Paging

```json theme={null}
{
  "module": "account",
  "count_only": true
}
```

## Contact Examples

### First Page of Contacts

```json theme={null}
{
  "module": "contact",
  "filters": {
    "fields": ["sfid", "firstname", "lastname", "email", "lastmodifieddate"],
    "order_by": "lastmodifieddate:DESC",
    "max_num": 50,
    "offset": 0
  }
}
```

### Next Page of Contacts

```json theme={null}
{
  "module": "contact",
  "filters": {
    "fields": ["sfid", "firstname", "lastname", "email", "lastmodifieddate"],
    "order_by": "lastmodifieddate:DESC",
    "max_num": 50,
    "offset": 50
  }
}
```

### Paginate Contacts with a Date Filter

```json theme={null}
{
  "module": "contact",
  "filters": {
    "fields": ["sfid", "firstname", "lastmodifieddate", "contact_type__c"],
    "order_by": "firstname:DESC",
    "max_num": 50,
    "offset": 0,
    "filter": [
      {
        "lastmodifieddate": {
          "$between": ["2025-01-01", "2025-01-20"]
        }
      }
    ]
  }
}
```

### Count Contacts Before Paging

```json theme={null}
{
  "module": "contact",
  "count_only": true
}
```

## Best Practices

* Prefer **`next_offset === -1`** as the primary stop condition for bulk jobs.
* Always include `order_by` so page results stay consistent across requests.
* Use a `max_num` between **20 and 50** for reliable bulk pulls.
* Use `count_only` first when you need the total record count before paging.
* Apply the same `filter` and `order_by` on every page of a bulk job.
* `max_num` works as either a number (`50`) or a string (`"50"`).
* Request fields use payload names (for example `sfid`, `lastname`); responses may return mapped names (for example `account_id` / `contact_id`, `last_name`).

## Token expiry

Access tokens expire after **18,000 seconds (5 hours)**.

When pulling large datasets (for example hundreds of thousands of Account or Contact records), a long-running pagination script can exceed this window. If the token expires mid-run, later page requests will fail with an unauthorized / invalid token response.

**Recommendations:**

* Track elapsed time while paging.
* Refresh or re-authenticate and obtain a new token before the 5-hour limit if the job may run longer.
* Resume from the last successful `offset` / `next_offset` after getting a new token so you do not restart from page 1.