A link in, a filed ad out
Someone pastes an Instagram post, a TikTok, or a Meta Ad Library link into the web app or the Telegram bot. The pipeline resolves share-sheet indirection links first — instagram.com/share/… and TikTok vm./vt. short links redirect to a canonical URL — then checks whether we already hold that exact ad. If we captured it within the last seven days, we re-link the existing copy and skip the fetch entirely: a re-share costs nothing, and only once engagement has drifted do we pay to refresh it.
The constraint that shapes everything: expiring signed URLs
When ScrapeCreators returns a post, the media comes back as signed CDN URLs that stop working — Instagram drops them after roughly four and a half days. That kills any naive caching: a cached snapshot is a list of links that mirror into nothing but 403s. Their cache_max_age is an enum — 1d, 3d, 7d, 14d, 30d — and 1d is the only value that fits under that expiry, so that is what we use. It is also why we download every asset into our own storage the moment we capture it: a creative only stays viewable because we copied it, not because anyone's URL is still alive.
Why we mirror before we serve
There is a second, subtler reason we copy media out of the source. Meta and Instagram serve creatives with a cross-origin resource policy that stops a browser rendering them from a different origin — so even a live URL is not enough. We download the image, the video and the thumbnail into object storage at ingest, guard against assets that come back blank or truncated (a reel will sometimes answer as a white ~600-byte JPEG), and serve everything from our own origin. An ad that fails to download is still kept — the capture is worth more than the tile.
Enrichment that never blocks a save
After an ad is stored, we classify it — niche, markets, topics, visual style, a one-line scene — and embed it for semantic search. That runs on a vision model, so it is deliberately not a gate: a classification outage must never fail someone's save. Untagged ads are simply swept up in the next batch. The result is a library you can ask what an ad is selling and who runs it, rather than one you can only scroll.