diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index d1ffaeb445..f390335ae9 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -11,3 +11,42 @@ Close # To simplify the testing workflow, please include the complete route, with all required and optional parameters, otherwise your pull request will be closed. --> + +## 新RSS检查列表 / New RSS Script Checklist + + + +- [ ] 这是在提交一个新的RSS吗? Is this a new RSS Script? + - **如果不是, 请留空本列表**. **LEAVE BLANK** if it is not a submitting new RSS Script +- [ ] 是否提供了文档? Documentation provided? + - [ ] 是否提供了英文文档? EN Documentation provided? +- [ ] 是否支持全文获取? Is this RSS Script support fulltext? + - [ ] 如果全文获取中需要访问文章链接, 是否使用了缓存? If fulltext requires to fetch detail pages, is cache used in the process? + - [缓存说明](https://docs.rsshub.app/joinus/#ti-jiao-xin-de-rsshub-gui-ze-bian-xie-jiao-ben-shi-yong-huan-cun) | [How to use cache](https://docs.rsshub.app/joinus/#ti-jiao-xin-de-rsshub-gui-ze-bian-xie-jiao-ben-shi-yong-huan-cun) +- [ ] 目标是否有明显的反爬/频率限制? Is there any sign of anti-bot or rate limit? + - [ ] 如果有, 是否有对应的措施? (延长缓存时间, 写文档说明, etc.) If yes, do your code reflect this sign? (e.g. write documentations, use long cache time) +- [ ] 是否引入的新的包? Any new package introduced? + - 如果有, 请说明原因. If yes, please state your reason +- [ ] 是否使用了`Puppeteer`? Make use of `Puppeteer`? + - 如果有, 请说明原因. If yes, please state your reason + + +## 说明 / Note + + diff --git a/assets/radar-rules.js b/assets/radar-rules.js index 47b97b2901..8eba207081 100644 --- a/assets/radar-rules.js +++ b/assets/radar-rules.js @@ -2500,4 +2500,18 @@ }, ], }, + 'scboy.com': { + _name: 'scboy 论坛', + www: [ + { + title: '帖子', + docs: 'https://docs.rsshub.app/bbs.html#scboy', + source: '', + target: (params, url) => { + const id = url.includes('thread') ? url.split('-')[1].split('.')[0] : ''; + return id ? `/scboy/thread/${id}` : ''; + }, + }, + ], + }, }); diff --git a/docs/bbs.md b/docs/bbs.md index 0a557cbee5..81779d9bf7 100644 --- a/docs/bbs.md +++ b/docs/bbs.md @@ -180,6 +180,17 @@ pageClass: routes +## SCBOY 论坛 + +### 帖子 + + + +帖子网址如果为 那么帖子 tid 就是 `1789863`。 + +访问水区需要添加环境变量 `SCBOY_BBS_TOKEN`, 详情见部署页面的配置模块。 `SCBOY_BBS_TOKEN`在 cookies 的`bbs_token`中。 + + ## V2EX ### 最热 / 最新主题 @@ -395,6 +406,24 @@ pageClass: routes +## 品葱 + +### 发现 + + + +| 最新 | 推荐 | 热门 | +| ---- | --------- | ---- | +| new | recommend | hot | + +### 精选 + + + +### 话题 + + + ## 三星盖乐世社区 ### 最新帖子 diff --git a/docs/en/install/README.md b/docs/en/install/README.md index 460be758ea..b612600623 100644 --- a/docs/en/install/README.md +++ b/docs/en/install/README.md @@ -104,6 +104,35 @@ $ docker run -d --name rsshub -p 1200:1200 -e CACHE_EXPIRE=3600 -e GITHUB_ACCESS To configure more options please refer to [Configuration](#configuration). +# Ansible Deployment + +This Ansible playbook includes RSSHub, Redis, browserless (uses Docker) and Caddy 2 + +Currently only support Ubuntu 20.04 + +Requires sudo privilege and virtualization capability (Docker will be automatically installed) + +### Install + +```bash +sudo apt update +sudo apt install ansible +git clone https://github.com/DIYgod/RSSHub.git ~/RSSHub +cd ~/RSSHub/scripts/ansible +sudo ansible-playbook rsshub.yaml +# When prompt to enter a domain name, enter the domain name that this machine/VM will use +# For example, if your users use https://rsshub.exmaple.com to access your RSSHub instance, enter rsshub.exmaple.com (remove the https://) +``` + +### Update + +```bash +cd ~/RSSHub/scripts/ansible +sudo ansible-playbook rsshub.yaml +# When prompt to enter a domain name, enter the domain name that this machine/VM will use +# For example, if your users use https://rsshub.exmaple.com to access your RSSHub instance, enter rsshub.exmaple.com (remove the https://) +``` + ## Manual Deployment The most direct way to deploy `RSSHub`, you can follow the steps below to deploy`RSSHub` on your computer, server or anywhere. diff --git a/docs/en/new-media.md b/docs/en/new-media.md index 070815cbe9..d0b372cc8b 100644 --- a/docs/en/new-media.md +++ b/docs/en/new-media.md @@ -67,6 +67,10 @@ Compared to the official one, the RSS feed generated by RSSHub not only has more ## CGTN +### Opinions + + + ### Most Read & Most Share diff --git a/docs/en/picture.md b/docs/en/picture.md index cd3cb4f024..f41db20042 100644 --- a/docs/en/picture.md +++ b/docs/en/picture.md @@ -42,6 +42,10 @@ pageClass: routes +## ComicsKingdom Comic Strips + + + ## DailyArt @@ -50,6 +54,10 @@ pageClass: routes +## GoComics Comic Strips + + + ## Google Doodles ### Update diff --git a/docs/en/program-update.md b/docs/en/program-update.md index 9325fd5e2b..13b7028215 100644 --- a/docs/en/program-update.md +++ b/docs/en/program-update.md @@ -164,6 +164,12 @@ The owner of the official image fills in the library, for example: https://rsshu +## Microsoft Store + +### Updates + + + ## Minecraft Refer to [#minecraft](/en/game.html#minecraft) diff --git a/docs/en/traditional-media.md b/docs/en/traditional-media.md index 983211fe80..7180f4bab0 100644 --- a/docs/en/traditional-media.md +++ b/docs/en/traditional-media.md @@ -20,6 +20,10 @@ Site ## AP News +### Top Stories + + + ### Topics @@ -149,6 +153,18 @@ Generates full-text feeds that the official feed doesn't provide. +## Radio Free Asia (RFA) + + + +Delivers a better experience by supporting parameter specification. + +Parameters can be obtained from the official website, for instance: + +`https://www.rfa.org/cantonese/news` corresponds to `/rfa/cantonese/news` + +`https://www.rfa.org/cantonese/news/htm` corresponds to `/rfa/cantonese/news/htm` + ## Reuters ### Channel diff --git a/docs/government.md b/docs/government.md index 7787edca28..eb76ada010 100644 --- a/docs/government.md +++ b/docs/government.md @@ -185,6 +185,18 @@ pageClass: routes +## 中国农工民主党 + +### 新闻中心 + + + +将目标栏目的网址拆解为 `http://www.ngd.org.cn/` 和后面的字段，去掉 `.htm` 后，把后面的字段中的 `/` 替换为 `-`，即为该路由的 slug + +如：(要闻动态)[http://www.ngd.org.cn/xwzx/ywdt/index.htm] 的网址在 `http://www.ngd.org.cn/` 后的字段是 `xwzx/ywdt/index.htm`，则对应的 slug 为 `xwzx-ywdt-index`，对应的路由即为 `/ngd/xwzx-ywdt-index` + + + ## 中国人大网 diff --git a/docs/install/README.md b/docs/install/README.md index 82c18370ac..a8c59fa918 100644 --- a/docs/install/README.md +++ b/docs/install/README.md @@ -106,6 +106,35 @@ $ docker run -d --name rsshub -p 1200:1200 -e CACHE_EXPIRE=3600 -e GITHUB_ACCESS 更多配置项请看 [#配置](#pei-zhi) +## Ansible 部署 + +这个 Ansible playbook 包括了 RSSHub, Redis, browserless (依赖 Docker) 以及 Caddy 2 + +目前只支持 Ubuntu 20.04 + +需要 sudo 权限和虚拟化能力（Docker 将会被自动安装） + +### 安装 + +```bash +sudo apt update +sudo apt install ansible +git clone https://github.com/DIYgod/RSSHub.git ~/RSSHub +cd ~/RSSHub/scripts/ansible +sudo ansible-playbook rsshub.yaml +# 当提示输入 domain name 的时候，输入该主机所使用的域名 +# 举例：如果您的 RSSHub 用户使用 https://rsshub.exmaple.com 访问您的 RSSHub 实例，输入 rsshub.exmaple.com（去掉 https://） +``` + +### 更新 + +```bash +cd ~/RSSHub/scripts/ansible +sudo ansible-playbook rsshub.yaml +# 当提示输入 domain name 的时候，输入该主机所使用的域名 +# 举例：如果您的 RSSHub 用户使用 https://rsshub.exmaple.com 访问您的 RSSHub 实例，输入 rsshub.exmaple.com（去掉 https://） +``` + ## 手动部署 部署 `RSSHub` 最直接的方式，您可以按照以下步骤将 `RSSHub` 部署在您的电脑、服务器或者其他任何地方 diff --git a/docs/live.md b/docs/live.md index f41ff9d02d..84f153c57d 100644 --- a/docs/live.md +++ b/docs/live.md @@ -22,7 +22,7 @@ pageClass: routes ### 直播分区 - + ::: warning 注意 diff --git a/docs/new-media.md b/docs/new-media.md index dea896d573..a9b2f164d2 100644 --- a/docs/new-media.md +++ b/docs/new-media.md @@ -122,6 +122,10 @@ pageClass: routes ## CGTN +### Opinions + + + ### Most Read & Most Share @@ -2028,6 +2032,9 @@ column 为 third 时可选的 category: +## 鱼塘热榜 + + ## 遠見 diff --git a/docs/program-update.md b/docs/program-update.md index 5cddf1086a..a78f659c95 100644 --- a/docs/program-update.md +++ b/docs/program-update.md @@ -212,6 +212,12 @@ pageClass: routes +## Microsoft Store + +### Updates + + + ## Minecraft 见 [#minecraft](/game.html#minecraft) @@ -310,6 +316,16 @@ pageClass: routes +## simpread + +### 消息通知 + + + +### 更新日志 + + + ## sketch.com ### beta 更新 diff --git a/docs/shopping.md b/docs/shopping.md index 2806418676..ad773c8220 100644 --- a/docs/shopping.md +++ b/docs/shopping.md @@ -172,6 +172,18 @@ For instance, in +## 麦当劳 + +### 麦当劳活动资讯 + + + +| 全部分类 | 社会责任 | 人员品牌 | 产品故事 | 优惠 | 品牌文化 | 活动速报 | +| --------- | -------------- | -------- | -------- | ----- | -------- | -------- | +| news_list | responsibility | brand | product | sales | culture | event | + + + ## 缺书网 ### 促销 diff --git a/docs/traditional-media.md b/docs/traditional-media.md index 94db156aeb..5fa926e34f 100644 --- a/docs/traditional-media.md +++ b/docs/traditional-media.md @@ -26,6 +26,10 @@ pageClass: routes ## AP News +### 首页头条 + + + ### 话题 @@ -181,6 +185,23 @@ pageClass: routes +## RTHK 傳媒透視 + + + +细则: + +- `:range` 时间范围参数 + (可为 `latest` 或 `四位数字的年份`) + + - `latest`: 最新的 50 篇文章 + - `2020`: 2020 年的所有文章 + +- 全文输出转换为简体字: `?opencc=t2s` + (`opencc` 是 RSSHub 的通用参数，详情请参阅 [「中文简繁体转换」](https://docs.rsshub.app/parameter.html#zhong-wen-jian-fan-ti-zhuan-huan)) + + + ## Solidot ### 最新消息 @@ -965,3 +986,15 @@ category 对应的关键词有 ### 九江新闻 + +## 自由亚洲电台 + + + +通过指定频道参数，提供比官方源更佳的阅读体验。 + +参数均可在官网获取，如： + +`https://www.rfa.org/cantonese/news` 对应 `/rfa/cantonese/news` + +`https://www.rfa.org/cantonese/news/htm` 对应 `/rfa/cantonese/news/htm` diff --git a/lib/config.js b/lib/config.js index 5e81e3e9e7..3e69c39790 100644 --- a/lib/config.js +++ b/lib/config.js @@ -185,6 +185,9 @@ const calculateValue = () => { username: envs.DIDA365_USERNAME, password: envs.DIDA365_PASSWORD, }, + scboy: { + token: envs.SCBOY_BBS_TOKEN, + }, }; }; calculateValue(); diff --git a/lib/router.js b/lib/router.js index 51da7962f9..ad181fc2e2 100644 --- a/lib/router.js +++ b/lib/router.js @@ -2491,6 +2491,7 @@ router.get('/gbcc/trust', require('./routes/gbcc/trust')); // Associated Press router.get('/apnews/topics/:topic', require('./routes/apnews/topics')); +router.get('/apnews', require('./routes/apnews/index')); // CBC router.get('/cbc/topics/:topic?', require('./routes/cbc/topics')); @@ -2853,6 +2854,9 @@ router.get('/xposed/module/:mod', require('./routes/xposed/module')); // Microsoft Edge router.get('/edge/addon/:crxid', require('./routes/edge/addon')); +// Microsoft Store +router.get('/microsoft-store/updates/:productid/:market?', require('./routes/microsoft-store/updates')); + // 上海立信会计金融学院 router.get('/slu/tzgg/:id', require('./routes/universities/slu/tzgg')); router.get('/slu/jwc/:id', require('./routes/universities/slu/jwc')); @@ -3339,6 +3343,7 @@ router.get('/fulinian/:caty?', require('./routes/fulinian/index')); // CGTN router.get('/cgtn/most/:type?/:time?', require('./routes/cgtn/most')); +router.get('/cgtn/opinions', require('./routes/cgtn/opinions')); // AppSales router.get('/appsales/:caty?/:time?', require('./routes/appsales/index')); @@ -3769,6 +3774,9 @@ router.get('/sciencenet/blog/:type?/:time?/:sort?', require('./routes/sciencenet // DailyArt router.get('/dailyart/:language?', require('./routes/dailyart/index')); +// SCBOY +router.get('/scboy/thread/:tid', require('./routes/scboy/thread')); + // 猿料 router.get('/yuanliao/:tag?/:sort?', require('./routes/yuanliao/index')); @@ -3781,6 +3789,37 @@ router.get('/nace/blog/:sort?', require('./routes/nace/blog')); // Caixin Latest router.get('/caixin/latest', require('./routes/caixin/latest')); +// 鱼塘热榜 +router.get('/mofish/:id', require('./routes/mofish/index')); + +// Mcdonalds +router.get('/mcdonalds/:category', require('./routes/mcdonalds/news')); + +// Pincong 品葱 +router.get('/pincong/category/:category?/:sort?', require('./routes/pincong/index')); +router.get('/pincong/hot/:category?', require('./routes/pincong/hot')); +router.get('/pincong/topic/:topic', require('./routes/pincong/topic')); + +// GoComics +router.get('/gocomics/:name', require('./routes/gocomics/index')); + +// Comics Kingdom +router.get('/comicskingdom/:name', require('./routes/comicskingdom/index')); + +// Media Digest +router.get('/mediadigest/:range/:category?', require('./routes/mediadigest/category')); + +// 中国农工民主党 +router.get('/ngd/:slug?', require('./routes/gov/ngd/index')); + +// SimpRead-消息通知 +router.get('/simpread/notice', require('./routes/simpread/notice')); +// SimpRead-更新日志 +router.get('/simpread/changelog', require('./routes/simpread/changelog')); + +// Radio Free Asia +router.get('/rfa/:language?/:channel?/:subChannel?', require('./routes/rfa/index')); + // booth.pm router.get('/booth.pm/shop/:subdomain', require('./routes/booth-pm/shop')); diff --git a/lib/routes/apnews/index.js b/lib/routes/apnews/index.js new file mode 100644 index 0000000000..a59c24f1de --- /dev/null +++ b/lib/routes/apnews/index.js @@ -0,0 +1,54 @@ +const cheerio = require('cheerio'); +const got = require('@/utils/got'); + +module.exports = async (ctx) => { + const response = await got.get('https://apnews.com/'); + const $ = cheerio.load(response.data); + const list = []; + + // main story component + const main = $('div[data-key=main-story]').first(); + + // headline news + const headline = {}; + headline.title = $(main).find('a[data-key=card-headline] h1').first().text(); + headline.link = 'https://apnews.com' + $(main).find('a[data-key=card-headline]').first().attr('href'); + list.push(headline); + + // related stories + $(main) + .find('a[data-key=related-story-link]') + .each(function (_, e) { + const item = {}; + item.title = $(e).find('div[data-key=related-story-headline]').first().text(); + item.link = 'https://apnews.com' + $(e).attr('href'); + list.push(item); + }); + + const result = await Promise.all( + list.map( + async (item) => + await ctx.cache.tryGet(item.link, async () => { + const content = await got.get(item.link); + + const description = cheerio.load(content.data); + const metadata = JSON.parse(description('script[type="application/ld+json"]').html()); + const featureImageURL = metadata.image; + + item.description = ``; + item.description += description('div[class=Article]') + .html() + .replace(/ADVERTISEMENT/g, ''); + item.pubDate = new Date(metadata.datePublished).toISOString(); + item.author = metadata.author[0]; + return item; + }) + ) + ); + + ctx.state.data = { + title: 'Associated Press News', + link: 'https://apnews.com/', + item: result, + }; +}; diff --git a/lib/routes/caixin/article.js b/lib/routes/caixin/article.js index 29fd6d70a0..6874fe8aad 100644 --- a/lib/routes/caixin/article.js +++ b/lib/routes/caixin/article.js @@ -1,4 +1,5 @@ const got = require('@/utils/got'); +const cheerio = require('cheerio'); module.exports = async (ctx) => { const response = await got({ @@ -12,15 +13,33 @@ module.exports = async (ctx) => { const data = response.data.data.list; + const items = await Promise.all( + data.map(async (item) => { + const link = item.web_url; + const summary = `

${item.summary}

`; + + const fullText = await ctx.cache.tryGet(link, async () => { + const result = await got.get(link); + + const $ = cheerio.load(result.data); + + return $('#Main_Content_Val').html(); + }); + + return { + title: item.title, + description: fullText ? summary + fullText : summary, + link: link, + pubDate: new Date(item.time * 1000), + author: item.author_name, + }; + }) + ); + ctx.state.data = { title: `财新网 - 首页`, link: `http://www.caixin.com/`, description: '财新网 - 首页', - item: data.map((item) => ({ - title: item.title, - description: `

${item.summary}

`, - link: item.web_url, - pubDate: new Date(item.time * 1000), - })), + item: items, }; }; diff --git a/lib/routes/cgtn/opinions.js b/lib/routes/cgtn/opinions.js new file mode 100644 index 0000000000..c10c3216c5 --- /dev/null +++ b/lib/routes/cgtn/opinions.js @@ -0,0 +1,51 @@ +const got = require('@/utils/got'); +const cheerio = require('cheerio'); + +module.exports = async (ctx) => { + const rootUrl = `https://www.cgtn.com/opinions`; + const response = await got({ + method: 'get', + url: rootUrl, + }); + + const $ = cheerio.load(response.data); + + $('.cg-pic').parent().remove(); + + const list = $(`.cg-title h4`) + .slice(0, 15) + .map((_, item) => { + item = $(item); + const a = item.find('a'); + return { + title: a.text(), + link: a.attr('href'), + pubDate: new Date(parseInt(a.attr('data-time'))).toUTCString(), + }; + }) + .get(); + + const items = await Promise.all( + list.map( + async (item) => + await ctx.cache.tryGet(item.link, async () => { + const detailResponse = await got({ + method: 'get', + url: item.link, + }); + const content = cheerio.load(detailResponse.data); + + item.author = content('.news-author-name').text(); + item.description = content('#cmsMainContent').html(); + + return item; + }) + ) + ); + + ctx.state.data = { + title: 'CGTN - Opinions', + link: rootUrl, + item: items, + }; +}; diff --git a/lib/routes/comicskingdom/index.js b/lib/routes/comicskingdom/index.js new file mode 100644 index 0000000000..3a81c7b47b --- /dev/null +++ b/lib/routes/comicskingdom/index.js @@ -0,0 +1,62 @@ +const got = require('@/utils/got'); +const cheerio = require('cheerio'); + +module.exports = async (ctx) => { + const baseURL = 'https://www.comicskingdom.com'; + const name = ctx.params.name; + const url = `${baseURL}/${name}/archive`; + const response = await got({ + method: 'get', + url: url, + }); + + const data = response.data; + const $ = cheerio.load(data); + + // Determine Comic and Author from main page + const comic = $('title').text().replace('Comics Kingdom - ', '').trim(); + const author = $('div.author p').text(); + + // Find the links for all non-archived items + const links = $('div.archive-tile[data-is-blocked=false]') + .map((i, el) => $(el).find('a[data-prem="Comic Tile"]').first().attr('href')) + .get() + .map((url) => `${baseURL}${url}`); + + if (links.length === 0) { + throw `Comic Not Found - ${name}`; + } + const items = await Promise.all( + links.map( + async (link) => + await ctx.cache.tryGet(link, async () => { + const detailResponse = await got({ + method: 'get', + url: link, + }); + const content = cheerio.load(detailResponse.data); + + const title = content('title').text(); + const image = content('meta[property="og:image"]').attr('content'); + const description = ``; + // Pull the date out of the URL + const pubDate = new Date(link.split('/').slice(-3).join('/')).toUTCString(); + + return { + title: title, + author: author, + category: 'comic', + description: description, + pubDate: pubDate, + link: link, + }; + }) + ) + ); + + ctx.state.data = { + title: `${comic} - Comics Kingdom`, + link: url, + item: items, + }; +}; diff --git a/lib/routes/douban/later.js b/lib/routes/douban/later.js index bf6a69ee0e..8d9b784ddd 100644 --- a/lib/routes/douban/later.js +++ b/lib/routes/douban/later.js @@ -1,19 +1,32 @@ const got = require('@/utils/got'); +const cheerio = require('cheerio'); module.exports = async (ctx) => { const response = await got({ method: 'get', - url: 'https://api.douban.com/v2/movie/coming_soon?apikey=0df993c66c0c636e29ecbb5344252a4a', + url: 'https://movie.douban.com/cinema/later/beijing/', }); - const movieList = response.data.subjects; + const $ = cheerio.load(response.data); + + const item = $('#showing-soon .item') + .map((index, ele) => { + const description = $(ele).html(); + const name = $('h3', ele).text().trim(); + const date = $('ul li', ele).eq(0).text().trim(); + const type = $('ul li', ele).eq(1).text().trim(); + const link = $('a.thumb', ele).attr('href'); + + return { + title: `${date} - 《${name}》 - ${type}`, + link, + description, + }; + }) + .get(); ctx.state.data = { title: '即将上映的电影', link: 'https://movie.douban.com/cinema/later/', - item: movieList.map((item) => ({ - title: item.title, - description: `标题：${item.title}
影片类型：${item.genres.join(' | ')}
评分：${item.rating.stars === '00' ? '无' : item.rating.average}
`, - link: item.alt, - })), + item, }; }; diff --git a/lib/routes/gocomics/index.js b/lib/routes/gocomics/index.js new file mode 100644 index 0000000000..b743418358 --- /dev/null +++ b/lib/routes/gocomics/index.js @@ -0,0 +1,64 @@ +const got = require('@/utils/got'); +const cheerio = require('cheerio'); + +module.exports = async (ctx) => { + const baseURL = 'https://www.gocomics.com'; + const name = ctx.params.name; + const limit = ctx.query.limit || 5; + const url = `${baseURL}/${name}/`; + const response = await got({ + method: 'get', + url: url, + }); + + const data = response.data; + const $ = cheerio.load(data); + + // Determine Comic and Author from main page + const comic = $('.media-heading').eq(0).text(); + const author = $('.media-subheading').eq(0).text().replace('By ', ''); + + // Load previous comic URL + const items = []; + let previous = $('.gc-deck--cta-0 a').attr('href'); + + if (!previous) { + throw `Comic Not Found - ${name}`; + } + + while (items.length < limit) { + const link = `${baseURL}${previous}`; + /* eslint-disable no-await-in-loop */ + const item = await ctx.cache.tryGet(link, async () => { + const detailResponse = await got({ + method: 'get', + url: link, + }); + const content = cheerio.load(detailResponse.data); + + const title = content('h1.m-0').eq(0).text(); + const image = content('.comic.container').eq(0).attr('data-image'); + const description = ``; + // Pull the date out of the URL + const pubDate = new Date(link.split('/').slice(-3).join('/')).toUTCString(); + const previous = content('.js-previous-comic').eq(0).attr('href'); + + return { + title: title, + author: author, + category: 'comic', + description: description, + pubDate: pubDate, + link: link, + previous: previous, + }; + }); + items.push(item); + previous = item.previous; + } + ctx.state.data = { + title: `${comic} - GoComics`, + link: url, + item: items, + }; +}; diff --git a/lib/routes/gov/ngd/index.js b/lib/routes/gov/ngd/index.js new file mode 100644 index 0000000000..5dfd996fe9 --- /dev/null +++ b/lib/routes/gov/ngd/index.js @@ -0,0 +1,52 @@ +const got = require('@/utils/got'); +const cheerio = require('cheerio'); + +module.exports = async (ctx) => { + const slug = ctx.params.slug || 'xwzx-ywdt-index'; + + const rootUrl = 'http://www.ngd.org.cn'; + const currentUrl = `${rootUrl}/${slug.replace(/-/g, '/')}.htm`; + const response = await got({ + method: 'get', + url: currentUrl, + }); + + const $ = cheerio.load(response.data); + + const list = $('.gp-ellipsis a') + .slice(0, 15) + .map((_, item) => { + item = $(item); + return { + title: item.text(), + link: `${currentUrl.replace('/index.htm', '')}/${item.attr('href')}`, + }; + }) + .get(); + + const items = await Promise.all( + list.map( + async (item) => + await ctx.cache.tryGet(item.link, async () => { + const detailResponse = await got({ + method: 'get', + url: item.link, + }); + const content = cheerio.load(detailResponse.data); + const info = content('.articleAuthor').text().split('|'); + + item.author = info[0].replace('来源：', ''); + item.description = content('.gp-article').html(); + item.pubDate = new Date(info[info.length - 1].replace('发布时间：', '')).toUTCString(); + + return item; + }) + ) + ); + + ctx.state.data = { + title: $('title').text(), + link: currentUrl, + item: items, + }; +}; diff --git a/lib/routes/idaily/index.js b/lib/routes/idaily/index.js index 3205f932a2..9c3567c3bd 100644 --- a/lib/routes/idaily/index.js +++ b/lib/routes/idaily/index.js @@ -6,19 +6,12 @@ module.exports = async (ctx) => { url: 'http://idaily-cdn.idailycdn.com/api/list/v3/iphone/zh-hans?page=1&ver=iphone', }); - const data = response.data; - - let dataToday = data.filter((item) => item.pubdate_timestamp * 1000 >= new Date().getTime() - 86400000); - - // leverage yesterday's items if today's not published yet - if (dataToday.length === 0) { - dataToday = data.filter((item) => item.pubdate_timestamp * 1000 >= new Date().getTime() - 86400000 * 2); - } + const data = response.data.filter((item) => item.ui_sets && item.ui_sets.caption_subtitle).slice(0, 15); ctx.state.data = { title: `iDaily 每日环球视野`, description: 'iDaily 每日环球视野', - item: dataToday.map((item) => ({ + item: data.map((item) => ({ title: item.ui_sets.caption_subtitle, description: `
${item.content}`, pubDate: new Date(item.pubdate_timestamp * 1000).toUTCString(), diff --git a/lib/routes/mcdonalds/news.js b/lib/routes/mcdonalds/news.js new file mode 100644 index 0000000000..f8fc8e3717 --- /dev/null +++ b/lib/routes/mcdonalds/news.js @@ -0,0 +1,53 @@ +const got = require('@/utils/got'); +const cheerio = require('cheerio'); + +module.exports = async (ctx) => { + const category_param = ctx.params.category || 'news_list'; + const categories = category_param.split('+'); + const baseUrl = 'https://www.mcdonalds.com.cn/news/'; + + const get_news_list = async (cates) => + await Promise.all( + cates.map(async (cate) => { + const response = await got.get(baseUrl + cate); + const $ = cheerio.load(response.data); + // console.log($('title').text()); + const news = $('.news_list .box-container > div') + .slice(0, 10) + .map((idx, item) => { + item = $(item); + return item.find('a[target]').attr('href'); + }) + .get(); + // console.log(news); + return Promise.resolve(news); + }) + ); + const all_news = [].concat.apply([], await get_news_list(categories)); + + const out = await Promise.all( + all_news.map(async (news_url) => { + const news_detail = await ctx.cache.tryGet(news_url, async () => { + const result = await got.get(news_url); + const $ = cheerio.load(result.data); + + // console.log($('h3 ~ time').text(), news_url); + + return { + title: $('h3').text(), + link: news_url, + // author: author, + pubDate: new Date($('h3 ~ time').text()).toUTCString(), + description: $('.cmsPage').html(), + }; + }); + return Promise.resolve(news_detail); + }) + ); + + ctx.state.data = { + title: '麦当劳资讯', + link: baseUrl, + item: out, + }; +}; diff --git a/lib/routes/mediadigest/category.js b/lib/routes/mediadigest/category.js new file mode 100644 index 0000000000..9c41c61b77 --- /dev/null +++ b/lib/routes/mediadigest/category.js @@ -0,0 +1,131 @@ +const got = require('@/utils/got'); +const cheerio = require('cheerio'); + +async function getArticle(ctx, list) { + const now = list.map( + async (line) => + await ctx.cache.tryGet(line, async () => { + const a_r = await got.get(`https://app3.rthk.hk/mediadigest/${line}`); + const $ = cheerio.load(a_r.data); + // title + const h1 = $('h1.story-title').text(); + // author + const author_list = $('div.story-author'); + const authors = author_list.map((_index, author) => $(author).text()); + const author = authors.map((_index, author) => `${author}
`); + const s_author = author.toArray().join(''); + const author_block = `

${s_author}

`; + // date + const date = $('div.story-calendar').text(); + // desc + const desc = `${$(author_block)}${$('div.story-content').html()}`; + + return { + title: h1, + description: desc, + pubDate: new Date(`${date}T09:00:00+0800`).toUTCString(), + link: line, + }; + }) + ); + // 執行一個切片 (20 個文章 URL) 的任務並將結果資料推至 rss 値 + return await Promise.all(now); +} + +module.exports = async (ctx) => { + const range = ctx.params.range || 'latest'; + const category = ctx.params.category || 'all'; + + // for range validation + const num = /^[1-9][0-9]{3}$/; + const current_year = new Date().getFullYear(); + // placeholders + let title = '傳媒透視'; + let rss = [ + { + description: '

Invalid :range input.

所輸入:range參數有誤。

所输入:range参数有误。

', + }, + ]; + // cid + const cids = [1, 2]; + for (let i = 4; i < 28; ++i) { + cids.push(i); + } + + // latest (50 articles): + // if 'all' (latest 50 articles of the site); + // else if 'cid' (deprecated) (latest 50 articles of a specific category); + // else 'error'. + if (range === 'latest') { + if (category === 'all') { + // 獲取全站文章 URL + const urls = cids.map((cid) => `https://app3.rthk.hk/mediadigest/category.php?cid=${cid}`); + const list_allt = await Promise.all( + urls.map(async (url) => { + const response = await got.get(url); + const $ = cheerio.load(response.data); + const list = $('div.category-line').map((_index, line) => $(line).find('a').first().attr('href')); + return Promise.resolve(list.toArray()); + }) + ); + const list_all = [...new Set(list_allt.flat())]; // removed duplicates and flatten + // 時序排列並抽取最新 50 項 + const list = list_all + .sort((a, b) => { + const aid_a = a.match(/aid=(\d+)/)[1]; + const aid_b = b.match(/aid=(\d+)/)[1]; + // reverse + return aid_b - aid_a; + }) + .slice(0, 50); + // getArticle(ctx, list); + rss = await getArticle(ctx, list); + } else if (cids.includes(parseInt(category))) { + // 獲取特定 category 文章目錄 + const response = await got.get(`https://app3.rthk.hk/mediadigest/category.php?cid=${category}`); + const $ = cheerio.load(response.data); + + const list = $('div.category-line') + .map((_index, line) => $(line).find('a').first().attr('href')) + .toArray(); + rss = await getArticle(ctx, list.slice(0, 50)); + } + } + // year (specific year range): + // if 'all' (latest 200 articles in a specific year range); + // else 'error'. + else if (num.test(range)) { + const range_num = parseInt(range); + if (category === 'all' && range_num >= 1970 && range_num <= current_year) { + // 獲取全站文章 URL + const urls = cids.map((cid) => `https://app3.rthk.hk/mediadigest/category.php?cid=${cid}`); + const list_all = await Promise.all( + // 每個任務是篩選出一個文章目錄裏需要抓取的文章 URL，以供後續「獲取全文」 + urls.map(async (url) => { + const response = await got.get(url); + const $ = cheerio.load(response.data); + // 對應文章 URL 與年份 + // Ref: https://cythilya.github.io/2016/03/13/jquery-map-grep/ + let list = $('div.category-line') + .map((_index, line) => $(line).find('a').first().attr('href')) + .toArray(); + const year = $('div.category-line div.pull-right') + .map((_index, date) => new Date($(date).text()).getFullYear()) + .toArray(); + list = list.filter((_url, i) => year[i] === range_num); + // 篩出 list (需要抓取的文章 URL 並組成數列) + return Promise.resolve(list); + }) + ); + const list = [...new Set(list_all.flat())]; // removed duplicates and flatten + rss = await getArticle(ctx, list.slice(0, 200)); + title = `傳媒透視 - ${range}`; + } + } + + ctx.state.data = { + title: title, + link: 'https://app3.rthk.hk/mediadigest/index.php', + item: rss, + }; +}; diff --git a/lib/routes/microsoft-store/updates.js b/lib/routes/microsoft-store/updates.js new file mode 100644 index 0000000000..4c5f63f625 --- /dev/null +++ b/lib/routes/microsoft-store/updates.js @@ -0,0 +1,32 @@ +const got = require('@/utils/got'); + +module.exports = async (ctx) => { + const { market = 'CN', productid } = ctx.params; + + const { data } = await got({ + method: 'get', + url: `https://displaycatalog.mp.microsoft.com/v7.0/products/${productid}/?fieldsTemplate=&market=${market}&languages=en`, + headers: { + 'Content-Type': 'application/json', + 'MS-CV': `${Array(16) + .join() + .split(',') + .map(function () { + return 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'.charAt(Math.floor(Math.random() * 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'.length)); + }) + .join('')}.1`, + }, + }); + + ctx.state.data = { + title: `${data.Product.LocalizedProperties[0].ProductTitle} - Microsoft Store Updates`, + link: `https://www.microsoft.com/store/productId/${productid}`, + item: [ + { + title: data.Product.DisplaySkuAvailabilities[0].Sku.Properties.Packages[0].PackageFullName, + pubDate: new Date(data.Product.DisplaySkuAvailabilities[0].Sku.LastModifiedDate), + link: `https://www.microsoft.com/store/productId/${productid}`, + }, + ], + }; +}; diff --git a/lib/routes/mofish/index.js b/lib/routes/mofish/index.js new file mode 100644 index 0000000000..e8890bf1ca --- /dev/null +++ b/lib/routes/mofish/index.js @@ -0,0 +1,29 @@ +const got = require('@/utils/got'); + +module.exports = async (ctx) => { + const id = ctx.params.id; + const page = ctx.query.page || 0; + + const url = `https://api.tophub.fun/v2/GetAllInfoGzip?id=${id}&page=${page}`; + + const response = await got({ + method: 'get', + url: url, + }); + + const data = response.data.Data.data; + + const title = `鱼塘热榜`; + + ctx.state.data = { + title: title, + link: `https://mo.fish/`, + description: title, + item: data.map((item) => ({ + title: item.Title, + pubDate: new Date(item.releaseTime).toUTCString(), + link: item.Url, + guid: item.id, + })), + }; +}; diff --git a/lib/routes/pincong/hot.js b/lib/routes/pincong/hot.js new file mode 100644 index 0000000000..0f65af2ab5 --- /dev/null +++ b/lib/routes/pincong/hot.js @@ -0,0 +1,33 @@ +const cheerio = require('cheerio'); + +module.exports = async (ctx) => { + let url = 'https://pincong.rocks/hot/list/'; + + url += ctx.params.category ? 'category-' + ctx.params.category : 'category-0'; + + // use Puppeteer due to the obstacle by cloudflare challenge + const browser = await require('@/utils/puppeteer')(); + const page = await browser.newPage(); + await page.goto(url); + const html = await page.evaluate( + // eslint-disable-next-line no-undef + () => document.documentElement.innerHTML + ); + + browser.close(); + + const $ = cheerio.load(html); + const list = $('div.aw-item'); + + ctx.state.data = { + title: '品葱 - 精选', + link: 'https://pincong.rocks/hot/', + item: list + .map((_, item) => ({ + title: $(item).find('h2 a').text().trim(), + description: $(item).find('div.markitup-box').html(), + link: 'https://pincong.rocks' + $(item).find('div.mod-head h2 a').attr('href'), + })) + .get(), + }; +}; diff --git a/lib/routes/pincong/index.js b/lib/routes/pincong/index.js new file mode 100644 index 0000000000..f1553de5f2 --- /dev/null +++ b/lib/routes/pincong/index.js @@ -0,0 +1,40 @@ +const cheerio = require('cheerio'); + +module.exports = async (ctx) => { + let url = 'https://pincong.rocks/'; + + const sortMap = { + new: 'sort_type-new', + recommend: 'recommend-1', + hot: 'sort_type-hot__day2', + }; + + url += (ctx.params.sort && sortMap[ctx.params.sort]) || 'recommend-1'; + url += ctx.params.category ? '__category-' + ctx.params.category : ''; + + // use Puppeteer due to the obstacle by cloudflare challenge + const browser = await require('@/utils/puppeteer')(); + const page = await browser.newPage(); + await page.goto(url); + const html = await page.evaluate( + // eslint-disable-next-line no-undef + () => document.querySelector('div.aw-common-list').innerHTML + ); + + browser.close(); + + const $ = cheerio.load(html); + const list = $('div.aw-item'); + + ctx.state.data = { + title: '品葱 - 发现', + link: url, + item: list + .map((_, item) => ({ + title: $(item).find('h4 a').text().trim(), + link: 'https://pincong.rocks' + $(item).find('h4 a').attr('href'), + pubDate: new Date($(item).attr('data-timestamp') * 1000).toISOString(), + })) + .get(), + }; +}; diff --git a/lib/routes/pincong/topic.js b/lib/routes/pincong/topic.js new file mode 100644 index 0000000000..2659515f80 --- /dev/null +++ b/lib/routes/pincong/topic.js @@ -0,0 +1,32 @@ +const cheerio = require('cheerio'); + +module.exports = async (ctx) => { + const url = 'https://pincong.rocks/topic/' + ctx.params.topic; + + // use Puppeteer due to the obstacle by cloudflare challenge + const browser = await require('@/utils/puppeteer')(); + const page = await browser.newPage(); + await page.goto(url); + const html = await page.evaluate( + () => + // eslint-disable-next-line no-undef + (document.querySelector('div.aw-common-list') && document.querySelector('div.aw-common-list').innerHTML) || '' + ); + + browser.close(); + + const $ = cheerio.load(html); + const list = $('div.aw-item'); + + ctx.state.data = { + title: `品葱 - ${ctx.params.topic}`, + link: url, + item: list + .map((_, item) => ({ + title: $(item).find('h4 a').text().trim(), + link: 'https://pincong.rocks' + $(item).find('h4 a').attr('href'), + pubDate: new Date($(item).attr('data-timestamp') * 1000).toISOString(), + })) + .get(), + }; +}; diff --git a/lib/routes/rfa/index.js b/lib/routes/rfa/index.js new file mode 100644 index 0000000000..90791eb2fc --- /dev/null +++ b/lib/routes/rfa/index.js @@ -0,0 +1,47 @@ +const cheerio = require('cheerio'); +const got = require('@/utils/got'); + +module.exports = async (ctx) => { + let url = 'https://www.rfa.org/' + (ctx.params.language || 'english'); + + if (ctx.params.channel) { + url += '/' + ctx.params.channel; + } + if (ctx.params.subChannel) { + url += '/' + ctx.params.subChannel; + } + + const response = await got.get(url); + const $ = cheerio.load(response.data); + + const selectors = ['div[id=topstorywidefulltease]', 'div.two_featured', 'div.three_featured', 'div.single_column_teaser', 'div.sectionteaser', 'div.specialwrap']; + const list = []; + selectors.forEach(function (selector) { + $(selector).each(function (_, e) { + const item = {}; + item.title = $(e).find('h2 a span').first().text(); + item.link = $(e).find('h2 a').first().attr('href'); + list.push(item); + }); + }); + + const result = await Promise.all( + list.map( + async (item) => + await ctx.cache.tryGet(item.link, async () => { + const content = await got.get(item.link); + + const description = cheerio.load(content.data); + item.description = description('div[id=storytext]').html(); + item.pubDate = new Date(description('span[id=story_date]').text()).toUTCString(); + return item; + }) + ) + ); + + ctx.state.data = { + title: 'RFA', + link: 'https://www.rfa.org/', + item: result, + }; +}; diff --git a/lib/routes/scboy/thread.js b/lib/routes/scboy/thread.js new file mode 100644 index 0000000000..acf87b25f8 --- /dev/null +++ b/lib/routes/scboy/thread.js @@ -0,0 +1,53 @@ +const got = require('@/utils/got'); +const cheerio = require('cheerio'); +const date = require('@/utils/date'); +const config = require('@/config').value; + +module.exports = async (ctx) => { + const tid = ctx.params.tid; + const link = `https://www.scboy.com/?thread-${tid}.htm`; + let cookieString = 'postlist_orderby=desc'; + if (config.scboy.token) { + cookieString = `postlist_orderby=desc; bbs_token=${config.scboy.token}`; + } + + const res = await got.get({ + method: 'get', + url: link, + responseType: 'buffer', + headers: { + Cookie: cookieString, + }, + }); + + const $ = cheerio.load(res.data); + const title = $('h4 > span:nth-of-type(1)').text(); + + const list = $('li.media.post'); + const count = []; + + for (let i = 0; i < Math.min(list.length, 30); i++) { + count.push(i); + } + + const resultItems = await Promise.all( + count.map(async (i) => { + const each = $(list[i]); + const floor = each.find('span.floor-parent').text(); + const item = { + title: `${title} #${floor ? floor : ' 热门回复'}`, + link: `https://www.scboy.com/?thread-${tid}`, + description: each.find('div.message.mt-1.break-all > div:nth-of-type(1)').html(), + author: each.find('username').text(), + pubDate: date(each.find('.date').text()), + }; + return Promise.resolve(item); + }) + ); + + ctx.state.data = { + title: title, + link: `https://www.scboy.com/?thread-${tid}.htm`, + item: resultItems, + }; +}; diff --git a/lib/routes/simpread/changelog.js b/lib/routes/simpread/changelog.js new file mode 100644 index 0000000000..59ecaea8f7 --- /dev/null +++ b/lib/routes/simpread/changelog.js @@ -0,0 +1,79 @@ +const got = require('@/utils/got'); +const cheerio = require('cheerio'); + +module.exports = async (ctx) => { + const url = 'http://ksria.com/simpread/changelog.html'; + const response = await got.get(url); + const data = response.data; + const $ = cheerio.load(data); + ctx.state.data = { + title: 'SimpRead 更新日志', + link: 'https://simpread.pro/changelog.html', + description: $('body > div.container.changelog > div.desc').html(), + item: $('.version') + .map((index, item) => { + const year = $(item).find('.year').html(); + const month_day = $(item).find('.day').html(); + let version = ''; + let detail = ''; + // 版本名称处理 + version = $(item).find('.num > a').clone().children().remove().end().text(); + // detail处理 + detail = $(item).find('.details'); + if (version === '') { + version = $(item).find('.num').clone().children().remove().end().text(); + } + let version_type = $(item).find('.num > a > i').attr('class'); + // 部分结构不一致处理 + if (version_type === undefined) { + version_type = $(item).find('.num > i').attr('class'); + } + if (version_type.indexOf('chrome') !== -1) { + version_type = 'Chrome'; + } + if (version_type.indexOf('apple') !== -1) { + version_type = 'Safari'; + } + if (version_type.indexOf('code') !== -1) { + version_type = 'UserScript'; + } + if (version_type.indexOf('firefox') !== -1) { + version_type = 'Firefox'; + } + + // detail + const text_color = { + important: '#9c27b0', + add: '#4caf50', + change: '#ffc107', + fix: '#f44336', + complete: '#03a9f4', + }; + $(detail) + .find('li') + .map((index, item) => { + let span_class = $(item).find('span').attr('class'); + const text = $(item).find('span').html(); + $(item).find('span').remove(); + if (span_class !== undefined) { + span_class = span_class.split(' '); + if (span_class[1] === 'empty') { + $(item).wrap('
    '); + } else { + $(item).prepend(`${text}: `); + } + } + return {}; + }); + + return { + description: $(detail).html(), + link: 'https://simpread.pro/changelog.html', + pubDate: `${year}-${month_day.replace('.', '-')} 00:00:00 GMT`, + title: `${version_type}${version}`, + author: 'SimpRead', + }; + }) + .get(), + }; +}; diff --git a/lib/routes/simpread/notice.js b/lib/routes/simpread/notice.js new file mode 100644 index 0000000000..79dd7950ef --- /dev/null +++ b/lib/routes/simpread/notice.js @@ -0,0 +1,20 @@ +const got = require('@/utils/got'); +const md = require('markdown-it')(); + +module.exports = async (ctx) => { + const url = 'https://static.simp.red/notice'; + const response = await got.get(url); + const data = response.data.notice; + ctx.state.data = { + title: 'SimpRead 消息通知', + link: 'https://simpread.pro/changelog.html', + description: 'SimpRead 消息通知', + item: data.map((item) => ({ + description: md.render(item.content), + link: 'https://simpread.pro/changelog.html', + pubDate: item.date, + title: `${item.category.name}-${item.title}`, + author: 'SimpRead', + })), + }; +}; diff --git a/lib/routes/universities/bit/cs/cs.js b/lib/routes/universities/bit/cs/cs.js index 605491fa6c..ab923eb31e 100644 --- a/lib/routes/universities/bit/cs/cs.js +++ b/lib/routes/universities/bit/cs/cs.js @@ -6,6 +6,9 @@ module.exports = async (ctx) => { const response = await got({ method: 'get', url: 'http://cs.bit.edu.cn/tzgg', + https: { + rejectUnauthorized: false, + }, }); const $ = cheerio.load(response.data); diff --git a/lib/routes/universities/bit/cs/utils.js b/lib/routes/universities/bit/cs/utils.js index d3d07dd1b6..2efa8cc240 100644 --- a/lib/routes/universities/bit/cs/utils.js +++ b/lib/routes/universities/bit/cs/utils.js @@ -5,7 +5,11 @@ const url = require('url'); // 专门定义一个function用于加载文章内容 async function load(link) { // 异步请求文章 - const response = await got.get(link); + const response = await got.get(link, { + https: { + rejectUnauthorized: false, + }, + }); // 加载文章内容 const $ = cheerio.load(response.data); diff --git a/lib/routes/universities/bit/jwc/jwc.js b/lib/routes/universities/bit/jwc/jwc.js index 175d6cdde3..52859cfe6f 100644 --- a/lib/routes/universities/bit/jwc/jwc.js +++ b/lib/routes/universities/bit/jwc/jwc.js @@ -6,6 +6,9 @@ module.exports = async (ctx) => { const response = await got({ method: 'get', url: 'http://jwc.bit.edu.cn/tzgg', + https: { + rejectUnauthorized: false, + }, }); const $ = cheerio.load(response.data); diff --git a/lib/routes/universities/bit/jwc/utils.js b/lib/routes/universities/bit/jwc/utils.js index 1c32dcfb67..d2891db294 100644 --- a/lib/routes/universities/bit/jwc/utils.js +++ b/lib/routes/universities/bit/jwc/utils.js @@ -5,7 +5,11 @@ const url = require('url'); // 专门定义一个function用于加载文章内容 async function load(link) { // 异步请求文章 - const response = await got.get(link); + const response = await got.get(link, { + https: { + rejectUnauthorized: false, + }, + }); // 加载文章内容 const $ = cheerio.load(response.data); diff --git a/lib/routes/universities/cpu/home.js b/lib/routes/universities/cpu/home.js index e7f1306d89..f87507ffc5 100644 --- a/lib/routes/universities/cpu/home.js +++ b/lib/routes/universities/cpu/home.js @@ -15,13 +15,13 @@ module.exports = async (ctx) => { const data = response.data; const $ = cheerio.load(data); - const $list = $('div#wp_news_w3 a').slice(0, 10).get(); + const $list = $('div#wp_news_w3 a').get(); const resultItem = await Promise.all( $list.map(async (item) => { const title = $(item).attr('title'); const href = $(item).attr('href'); - const detail_url = 'http://www.cpu.edu.cn' + href; + const detail_url = href.startsWith('/') ? `http://www.cpu.edu.cn${href}` : href; const single = { title: title, link: detail_url, @@ -37,7 +37,7 @@ module.exports = async (ctx) => { { const detail_data = detail.data; const $ = cheerio.load(detail_data); - single.description = $('table[bgcolor="#FFFFFF"]').html(); + single.description = $('table[bgcolor="#FFFFFF"]').html() || $('table.nyxx').html() || $('div.inner div.article').html(); } return Promise.resolve(single); }) diff --git a/lib/routes/xiaoheihe/news.js b/lib/routes/xiaoheihe/news.js index 1eb2d2f827..a00a6730bf 100644 --- a/lib/routes/xiaoheihe/news.js +++ b/lib/routes/xiaoheihe/news.js @@ -38,6 +38,7 @@ module.exports = async (ctx) => { // 存放到缓存区 ctx.cache.set(cacheKey, content); news.description = content; + news.link = `https://api.xiaoheihe.cn/maxnews/app/share/detail/${newsId}`; } return Promise.resolve(news); diff --git a/scripts/ansible/.gitignore b/scripts/ansible/.gitignore new file mode 100644 index 0000000000..6117c52186 --- /dev/null +++ b/scripts/ansible/.gitignore @@ -0,0 +1,90 @@ +# Created by https://www.toptal.com/developers/gitignore/api/windows,linux,macos,ansible,vagrant +# Edit at https://www.toptal.com/developers/gitignore?templates=windows,linux,macos,ansible,vagrant + +### Ansible ### +*.retry + +### Linux ### +*~ + +# temporary files which can be created if a process still has a handle open of a deleted file +.fuse_hidden* + +# KDE directory preferences +.directory + +# Linux trash folder which might appear on any partition or disk +.Trash-* + +# .nfs files are created when an open file is removed but is still being accessed +.nfs* + +### macOS ### +# General +.DS_Store +.AppleDouble +.LSOverride + +# Icon must end with two \r +Icon + + +# Thumbnails +._* + +# Files that might appear in the root of a volume +.DocumentRevisions-V100 +.fseventsd +.Spotlight-V100 +.TemporaryItems +.Trashes +.VolumeIcon.icns +.com.apple.timemachine.donotpresent + +# Directories potentially created on remote AFP share +.AppleDB +.AppleDesktop +Network Trash Folder +Temporary Items +.apdisk + +### Vagrant ### +# General +.vagrant/ + +# Log files (if you are creating logs in debug mode, uncomment this) +# *.log + +### Vagrant Patch ### +*.box + +### Windows ### +# Windows thumbnail cache files +Thumbs.db +Thumbs.db:encryptable +ehthumbs.db +ehthumbs_vista.db + +# Dump file +*.stackdump + +# Folder config file +[Dd]esktop.ini + +# Recycle Bin used on file shares +$RECYCLE.BIN/ + +# Windows Installer files +*.cab +*.msi +*.msix +*.msm +*.msp + +# Windows shortcuts +*.lnk + +# End of https://www.toptal.com/developers/gitignore/api/windows,linux,macos,ansible,vagrant + +# vagrant logs +*.log \ No newline at end of file diff --git a/scripts/ansible/README.md b/scripts/ansible/README.md new file mode 100644 index 0000000000..50cae9b717 --- /dev/null +++ b/scripts/ansible/README.md @@ -0,0 +1,20 @@ +# Readme + +Ansible playbook to deploy [RSSHub](https://github.com/DIYgod/RSSHub) on bare-metal with Redis, browserless and Caddy 2 + +Requires sudo permission + +## Usage +On `Ubuntu 20.04`, [install ansible](https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-ansible-on-ubuntu-20-04), then + +```bash +sudo ansible-playbook rsshub.yaml +``` + +## Development +Install `vagrant`, then + +```bash +./try.sh +ansible-playbook rsshub.yaml +``` diff --git a/scripts/ansible/Vagrantfile b/scripts/ansible/Vagrantfile new file mode 100644 index 0000000000..604af7f5ad --- /dev/null +++ b/scripts/ansible/Vagrantfile @@ -0,0 +1,5 @@ +Vagrant.configure("2") do |config| + config.vm.box = "generic/ubuntu2004" + config.vm.synced_folder ".", "/vagrant", type: "rsync", rsync__exclude: ".git/" + config.ssh.extra_args = ["-t", "cd /vagrant; bash --login"] +end diff --git a/scripts/ansible/rsshub.Caddyfile b/scripts/ansible/rsshub.Caddyfile new file mode 100644 index 0000000000..7781eb24fd --- /dev/null +++ b/scripts/ansible/rsshub.Caddyfile @@ -0,0 +1,3 @@ +{{ domain_name }} + +reverse_proxy localhost:1200 diff --git a/scripts/ansible/rsshub.env b/scripts/ansible/rsshub.env new file mode 100644 index 0000000000..d4a283725c --- /dev/null +++ b/scripts/ansible/rsshub.env @@ -0,0 +1,3 @@ +NODE_ENV=production +CACHE_TYPE=redis +PUPPETEER_WS_ENDPOINT=ws://localhost:3000 diff --git a/scripts/ansible/rsshub.service b/scripts/ansible/rsshub.service new file mode 100644 index 0000000000..0c8352a44e --- /dev/null +++ b/scripts/ansible/rsshub.service @@ -0,0 +1,11 @@ +[Unit] +Description=RSSHub is an open source, easy to use, and extensible RSS feed aggregator + +[Service] +User=rsshub +WorkingDirectory=/home/rsshub/app +ExecStart=yarn start +EnvironmentFile=/home/rsshub/app/.env + +[Install] +WantedBy=multi-user.target diff --git a/scripts/ansible/rsshub.yaml b/scripts/ansible/rsshub.yaml new file mode 100644 index 0000000000..33671a51ce --- /dev/null +++ b/scripts/ansible/rsshub.yaml @@ -0,0 +1,130 @@ +- + name: Install RSSHub + hosts: localhost + become: true + vars_prompt: + - + name: domain_name + prompt: What is the domain name (without www, e.g. rsshub.example.com)? Use "http://localhost" for development in Vagrant VM. + private: no + tasks: + - + name: Check OS + fail: + msg: This playbook can only be run on Ubuntu 20.04 at this moment + when: ansible_distribution != 'Ubuntu' or ansible_distribution_version !='20.04' + - + name: Install GPG keys for repos + apt_key: + url: '{{ item }}' + state: present + with_items: + - https://deb.nodesource.com/gpgkey/nodesource.gpg.key + - https://dl.yarnpkg.com/debian/pubkey.gpg + - https://download.docker.com/linux/ubuntu/gpg + - https://dl.cloudsmith.io/public/caddy/stable/cfg/gpg/gpg.155B6D79CA56EA34.key + - + name: Install repos + apt_repository: + repo: '{{ item }}' + state: present + update_cache: yes + with_items: + - deb https://deb.nodesource.com/node_12.x focal main + - deb https://dl.yarnpkg.com/debian/ stable main + - deb https://download.docker.com/linux/ubuntu focal stable + - deb https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version main + - + name: Install prerequisites + apt: + name: + - nodejs + - yarn + - build-essential + - python-is-python2 + - redis-server + - docker-ce + - python3-pip + - virtualenv + - python3-setuptools + - caddy + state: present + update_cache: yes + - + name: Install python module for docker + pip: + name: docker + - + name: Pull docker image for browserless + docker_image: + name: browserless/chrome + source: pull + - + name: Start redis + systemd: + state: restarted + enabled: yes + name: redis + daemon_reload: yes + - + name: Copy caddy configuration + template: + src: rsshub.Caddyfile + dest: /etc/caddy/Caddyfile + - + name: Start caddy + systemd: + state: restarted + enabled: yes + name: caddy + daemon_reload: yes + - + name: Create and start browserless container + docker_container: + name: browserless + image: browserless/chrome + state: started + restart_policy: always + published_ports: + - "3000:3000" + - + name: Create the user + user: + name: rsshub + create_home: true + shell: /bin/bash + - + name: Clone the repo + git: + repo: https://github.com/DIYgod/RSSHub.git + dest: /home/rsshub/app + update: yes + - + name: Install repo dependencies + command: yarn install --production + args: + chdir: /home/rsshub/app + - + name: Copy configuration + copy: + src: rsshub.env + dest: /home/rsshub/app/.env + - + name: Own repo to the user + file: + path: /home/rsshub/app + owner: rsshub + group: rsshub + recurse: yes + - + name: Install the systemd unit + copy: + src: rsshub.service + dest: /etc/systemd/system/rsshub.service + - + name: Start the systemd service + systemd: + state: restarted + enabled: yes + name: rsshub + daemon_reload: yes diff --git a/scripts/ansible/try.sh b/scripts/ansible/try.sh new file mode 100755 index 0000000000..cd8f1df76d --- /dev/null +++ b/scripts/ansible/try.sh @@ -0,0 +1,5 @@ +#!/bin/bash +set -e + +vagrant rsync +vagrant ssh