Export mentions using streams

What is Streaming API?

Streaming API provides long-running connections that continuously deliver mentions as they match your filters. Unlike paginated search that returns 25 mentions per request, streaming API can deliver unlimited volumes of data through a single connection.

Why use Streaming API?

• Bulk data export - Extract large datasets efficiently without pagination
• Real-time processing - Process mentions as they arrive in your system
• Reduced API calls - One stream connection vs. hundreds of paginated requests
• Better performance - Continuous data flow without request overhead

Key Differences

Historical Mentions Export Stream

Follow these steps to export historical mentions from Listen project using streaming API:

1. List Projects

curl -X GET \
  'https://sentione.com/api/public/v2/projects' \
  -H 'X-API-KEY: your_secret_key_here'

2. Extract project ID from the response

{
  "data": [
    { "id": 111, "name": "Apple" },
    { "id": 222, "name": "Android" }
  ],
  "cursor": "abc123"
}

Select desired project id (eg. 111 for Apple project)

3. Start historical mentions export stream

curl -N -X POST \
  'https://sentione.com/api/public/v2/projects/111/mentions/search/stream' \
  -H 'X-API-KEY: your_secret_key_here' \
  -H 'Content-Type: application/json' \
  -H 'Accept: text/event-stream' \
  -d '{
    "filters": {
      "publishedAt": {
        "from": "2025-01-01T00:00:00Z",
        "to": "2025-05-22T23:59:59Z"
      }
    },
    "sortType": "PublishedAtDescending"
  }'

4. Process streaming events

The stream will return Server-Sent Events (SSE) format:

event: mention
data: {"id": "smid-wcuEd-YCoP89RC676Z97taQ2Nk97_T5HYBbFSb1rwOCzvdVLUm5GMI6tZXFUhoU5v0zQiWgt48ZllN5iXPF9tVuU4dFDR0COkaoLAVFOrYCbXju8Nztir6cPSOMdITnpMZMjNsALWcK3MU1tevSnNpZojdQD2IzBQfzYx_o", "author": {"id": "333333", "name": "Apple user", "avatar": {"url": "https://graph.facebook.com/1111111111/picture", "type": "Image"}, "gender": "Unspecified", "influenceMetrics": {}}, "source": {"socialChannel": {"id": "1111111111", "name": "Apple iPhone Fanpage Example"}, "mentionUrl": "https://www.facebook.com/1111111111/", "type": "Facebook"}, "type": "Post", "publishedAt": "2025-03-15T14:22:31.000Z", "collectedAt": "2025-03-15T14:22:36.122Z", "content": {"text": "Apple is revolutionary!", "language": {"code": "en"}, "sentiment": "Positive"}, "engagementMetrics": {"influenceScore": 8, "engagementRate": 0.125, "sourceMetrics": {"common": {"likeCount": 45, "shareCount": 12, "commentCount": 8}, "facebook": {"likeCount": 45, "loveCount": 15, "hahaCount": 2, "wowCount": 3, "sadCount": 0, "angryCount": 1}}}, "attachments": []}
id: smid-wcuEd-YCoP89RC676Z97taQ2Nk97_T5HYBbFSb1rwOCzvdVLUm5GMI6tZXFUhoU5v0zQiWgt48ZllN5iXPF9tVuU4dFDR0COkaoLAVFOrYCbXju8Nztir6cPSOMdITnpMZMjNsALWcK3MU1tevSnNpZojdQD2IzBQfzYx_o

event: heartbeat
data: {"date": "2025-05-22T10:30:00Z"}
id: 2025-05-22T10:30:00Z

event: mention
data: {"id": "smid-another-mention-id", "author": {"id": "444444", "name": "Tech Reviewer", "gender": "Male"}, "source": {"socialChannel": {"id": "2222222222", "name": "Tech Blog"}, "mentionUrl": "https://techblog.com/apple-review", "type": "Websites"}, "type": "Article", "publishedAt": "2025-03-14T09:15:22.000Z", "collectedAt": "2025-03-14T09:16:01.445Z", "content": {"text": "Apple's latest product shows mixed results in our tests.", "language": {"code": "en"}, "sentiment": "Neutral"}, "engagementMetrics": {"influenceScore": 9, "engagementRate": 0.089}, "attachments": []}
id: smid-another-mention-id

5. Alternative: JSON streaming format

For JSON lines format instead of SSE:

curl -N -X POST \
  'https://sentione.com/api/public/v2/projects/111/mentions/search/stream' \
  -H 'X-API-KEY: your_secret_key_here' \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -d '{
    "filters": {
      "publishedAt": {
        "from": "2025-01-01T00:00:00Z",
        "to": "2025-05-22T23:59:59Z"
      }
    },
    "sortType": "PublishedAtDescending"
  }'

This returns newline-delimited JSON:

{"event": "mention", "data": {...}, "id": "mention_id"}
{"event": "heartbeat", "data": {"date": "2025-05-22T10:30:00Z"}, "id": "2025-05-22T10:30:00Z"}
{"event": "mention", "data": {...}, "id": "another_mention_id"}

Recent Mentions Export Stream

For recent mentions (last 7 days only), use the recent export stream:

curl -N -X POST \
  'https://sentione.com/api/public/v2/projects/111/mentions/recent/search/stream' \
  -H 'X-API-KEY: your_secret_key_here' \
  -H 'Content-Type: application/json' \
  -H 'Accept: text/event-stream' \
  -d '{
    "filters": {
      "content": {
        "sentiment": ["Positive", "Negative"]
      },
      "source": {
        "type": ["Facebook", "Instagram", "X"]
      }
    },
    "sortType": "PublishedAtDescending"
  }'

Stream Processing Tips

1. Use non-blocking event processing - Process events asynchronously to avoid blocking the stream. If your processing logic is slow, you may not keep up with incoming data and lose mentions.
2. Handle heartbeat events to maintain connection health
3. Process events incrementally - don't wait for stream completion
4. Implement reconnection logic for production use
5. Monitor rate limits - only 2 concurrent stream connections allowed per account
6. Choose appropriate format - SSE for real-time, JSON for batch processing

For full details on endpoints, schemas, filters, sorting options, and time ranges, refer to the OpenAPI documentation.