RSS Feeds¶
The RSS archiver is a generic source that works with any standard RSS or Atom feed.
Configuration¶
[sources.hackaday]
type = "rss"
display_name = "Hackaday"
frequency = "daily"
enabled = true
url = "http://www.hackaday.com/rss.xml"
max_articles = 30
include_article_content = true
Options¶
| Option | Type | Default | Description |
|---|---|---|---|
type |
string | — | Must be "rss" |
display_name |
string | — | Name shown in output |
frequency |
string | "daily" |
Fetch frequency hint |
enabled |
bool | true |
Enable/disable this source |
url |
string | — | URL of the RSS/Atom feed |
max_articles |
int | 30 |
Maximum articles to fetch |
include_article_content |
bool | true |
Fetch full article content from linked pages |
How It Works¶
- Fetches and parses the RSS/Atom feed using
feedparser - Extracts article metadata from feed entries (title, URL, summary, date)
- If
include_article_contentistrue, visits each article's URL and extracts readable content using readability - Downloads run concurrently using the configured
max_workers
Adding Multiple RSS Sources¶
You can add as many RSS sources as you want — just give each a unique name:
[sources.hackaday]
type = "rss"
display_name = "Hackaday"
enabled = true
url = "http://www.hackaday.com/rss.xml"
max_articles = 30
include_article_content = true
[sources.engadget]
type = "rss"
display_name = "Engadget"
enabled = true
url = "https://www.engadget.com/rss.xml"
max_articles = 30
include_article_content = true
[sources.ars_technica]
type = "rss"
display_name = "Ars Technica"
enabled = true
url = "https://feeds.arstechnica.com/arstechnica/index"
max_articles = 20
include_article_content = true
Tips¶
Feed Discovery
Most websites expose their RSS feed at /rss.xml, /feed, or /atom.xml. Browser extensions like "Get RSS Feed URL" can help find feeds.
Content Extraction
Set include_article_content = false if the feed already contains full content in its entries, or if you only want headlines and summaries.
Rate Limiting
Some websites may rate-limit or block automated requests. If you encounter issues, try reducing max_articles or max_workers.