Docs



Using the Cursor Parameter in the API


The cursor parameter is an essential tool for managing paginated data retrieval when working with specific API endpoints. This parameter allows you to navigate through large datasets by specifying the position within a list of results. The cursor-based pagination is implemented in the following endpoints:


  • search.php: Retrieves search results based on your query.
  • timeline.php: Fetches a user's timeline data.
  • followers.php: Retrieves the list of followers of a specified user.
  • following.php: Fetches the list of accounts a user is following.
  • replies.php: Retrieves replies to a specific tweet or user.
  • usermedia.php: Fetches media associated with a specific user.

How the Cursor Parameter Works

When you make a request to any of these endpoints, the API response will include a field in the JSON body called "next_cursor". This value serves as a pointer to the next set of results in the paginated list.


Basic Workflow:

  1. Initial Request: Make a request without the cursor parameter to retrieve the first page of results.
  2. Fetch Next Page: Use the next_cursor value from the response as the cursor parameter in your next API request.
  3. Repeat: Continue using the updated next_cursor value from subsequent responses until the "next_cursor" field is empty or null, indicating the end of the dataset.

Example Workflow


Step 1: Initial Request

Make a request to an endpoint without the cursor parameter. For example:

curl -X GET "https://api.scraper.tech/followers.php?screenname=elonmusk" \
-H "Scraper-Key: YOUR_UNIQUE_KEY"

Response:

{
  "followers": [
    {"id": 1, "name": "John Doe"},
    {"id": 2, "name": "Jane Smith"}
  ],
  "next_cursor": "abc123"
}

Step 2: Fetch the Next Page

Use the value of "next_cursor" in your subsequent request:

curl -X GET "https://api.scraper.tech/followers.php?screenname=elonmusk&cursor=abc123" \
-H "Scraper-Key: YOUR_UNIQUE_KEY"

Response:

{
  "followers": [
    {"id": 3, "name": "Alice Brown"},
    {"id": 4, "name": "Bob White"}
  ],
  "next_cursor": "def456"
}

Step 3: Repeat Until Complete

Repeat the process with the updated cursor parameter until the "next_cursor" field is empty or null.

{
  "followers": [],
  "next_cursor": null
}

Tips for Using the Cursor Parameter

  • Handle Empty Cursors Gracefully: Always check if the "next_cursor" field is null before making the next request.
  • Rate Limits: Be aware of rate limits for each endpoint and pace your requests accordingly.
  • Store Progress: Save the cursor value after each request to ensure continuity in case of errors or interruptions.
  • Avoid Overloading: Fetch data in manageable chunks to reduce the risk of hitting API limits or encountering server timeouts.

By using the cursor parameter effectively, you can retrieve large datasets efficiently and maintain seamless pagination across multiple API endpoints.