fix(route): ABC News (#8217)

This commit is contained in:
Ethan Shen
2021-09-22 21:47:15 -07:00
committed by GitHub
parent 2fa51b5c2d
commit ee5a81be09
6 changed files with 14674 additions and 65 deletions
+45 -7
View File
@@ -6,17 +6,55 @@ pageClass: routes
## ABC News
### Site
### Channel & Topic
<Route author="nczitzk" example="/abc" path="/abc/:site?" :paramsDesc="['Site, see below']">
<RouteEn author="nczitzk" example="/abc" path="/abc/:id?" :paramsDesc="['id, can be found in the Channel or Topic page, can also be filled in the `documentId` in the source code of the page, see below, Just In by default']">
Site
::: tip Tip
| Just In | Politics | World | Business | Analysis | Sport | Science | Health | Arts | Fact Check | 中文新闻 | Berita Bahasa Indonesia | Tok Pisin |
| - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| justin | politics | world | business | analysis-and-opinion | sport | science | health | arts-culture | factcheck | chinese | indonesian | tok-pisin |
All Topics in [Topic Library](https://abc.net.au/news/topics) are supported, you can fill in the field after `topic` in its URL, or fill in the `documentId`.
</Route>
For example, the URL for [Computers and Technology](https://www.abc.net.au/news/topic/computers-and-technology) is <https://www.abc.net.au/news/topic/computers-and-technology>, the field after `topic` is `computers-and-technology`, and the `documentId` of the Topic is `2302`, so the route is [/abc/computers-and-technology](https://rsshub.app/abc/computers-and-technology) and [/abc/2302](https://rsshub.app/abc/2302).
The supported channels are all listed in the table below. For other channels, please find the `documentId` in the source code of the channel page and fill it in as above.
:::
The following are some of the supported Channel and Topic ids.
| Just In | Coronavirus | Politics | World |
| ------- | ----------- | -------- | ----- |
| justin | coronavirus | politics | world |
| Asia Pacific | Business | Analysis & Opinion | Sport |
| ------------ | -------- | -------------------- | ----- |
| asia-pacific | business | analysis-and-opinion | sport |
| AFL | Rugby League | Rugby Union | Football |
| --- | ------------ | ----------- | -------- |
| afl | rugbyleague | rugbyunion | football |
| Cricket | Science | Astronomy (Space) | Computers and Technology |
| ------- | ------- | ----------------- | ------------------------ |
| cricket | science | astronomy-space | computers-and-technology |
| Environment | Archaeology | Health | Exercise and Fitness |
| ----------- | ----------- | ------ | -------------------- |
| environment | archaeology | health | exercise-and-fitness |
| Pharmaceuticals | Mental Health | Diet and Nutrition | Arts & Culture |
| --------------- | ------------- | ------------------ | -------------- |
| pharmaceuticals | mental-health | diet-and-nutrition | arts-culture |
| Fact Check | ABC 中文 | 澳洲时政 | 聚焦中港台 |
| ---------- | -------- | ------------------ | -------------- |
| factcheck | chinese | australia-politics | focus-on-china |
| 观点与分析 | 澳洲华人 | 解读澳洲 | Berita dalam Bahasa Indonesia | Tok Pisin |
| ----------------------- | -------------------- | ----------------- | ----------------------------- | --------- |
| analysis-and-opinion-zh | chinese-in-australia | curious-australia | indonesian | tok-pisin |
</RouteEn>
## AP News
+44 -6
View File
@@ -12,15 +12,53 @@ pageClass: routes
## ABC News
### 子站
### Channel & Topic
<Route author="nczitzk" example="/abc/chinese" path="/abc/:site?" :paramsDesc="['子站,见下表']">
<Route author="nczitzk" example="/abc" path="/abc/:id?" :paramsDesc="['id,可在对应 Channel 或 Topic 页面中找到,也可以填入对应页源代码中的 `documentId`,部分见下表,默认为 Just In']">
子站
::: tip 提示
| Just In | Politics | World | Business | Analysis | Sport | Science | Health | Arts | Fact Check | 中文新闻 | Berita Bahasa Indonesia | Tok Pisin |
| ------- | -------- | ----- | -------- | -------------------- | ----- | ------- | ------ | ------------ | ---------- | -------- | ----------------------- | --------- |
| justin | politics | world | business | analysis-and-opinion | sport | science | health | arts-culture | factcheck | chinese | indonesian | tok-pisin |
支持 [Topic Library](https://abc.net.au/news/topics) 中的所有 Topic,你可以填入其 URL 中 `topic` 后的字段,也可以填入对应的 `documentId`。
如 [Computers and Technology](https://www.abc.net.au/news/topic/computers-and-technology) 的 URL 是 <https://www.abc.net.au/news/topic/computers-and-technology>,其 `topic` 后的字段为 `computers-and-technology`,且该 Topic 的 `documentId` 为 `2302`,所以路由为 [/abc/computers-and-technology](https://rsshub.app/abc/computers-and-technology) 以及 [/abc/2302](https://rsshub.app/abc/2302)。
支持的 Channel 在下表已经全部列出,同上,其他 Channel 请找到对应 Channel 页面的源代码中的 `documentId` 填入。
:::
以下是部分支持 Channel 和 Topic id:
| Just In | Coronavirus | Politics | World |
| ------- | ----------- | -------- | ----- |
| justin | coronavirus | politics | world |
| Asia Pacific | Business | Analysis & Opinion | Sport |
| ------------ | -------- | -------------------- | ----- |
| asia-pacific | business | analysis-and-opinion | sport |
| AFL | Rugby League | Rugby Union | Football |
| --- | ------------ | ----------- | -------- |
| afl | rugbyleague | rugbyunion | football |
| Cricket | Science | Astronomy (Space) | Computers and Technology |
| ------- | ------- | ----------------- | ------------------------ |
| cricket | science | astronomy-space | computers-and-technology |
| Environment | Archaeology | Health | Exercise and Fitness |
| ----------- | ----------- | ------ | -------------------- |
| environment | archaeology | health | exercise-and-fitness |
| Pharmaceuticals | Mental Health | Diet and Nutrition | Arts & Culture |
| --------------- | ------------- | ------------------ | -------------- |
| pharmaceuticals | mental-health | diet-and-nutrition | arts-culture |
| Fact Check | ABC 中文 | 澳洲时政 | 聚焦中港台 |
| ---------- | -------- | ------------------ | -------------- |
| factcheck | chinese | australia-politics | focus-on-china |
| 观点与分析 | 澳洲华人 | 解读澳洲 | Berita dalam Bahasa Indonesia | Tok Pisin |
| ----------------------- | -------------------- | ----------------- | ----------------------------- | --------- |
| analysis-and-opinion-zh | chinese-in-australia | curious-australia | indonesian | tok-pisin |
</Route>
+1 -1
View File
@@ -3413,7 +3413,7 @@ router.get('/icity/:id', lazyloadRouteHandler('./routes/icity/index'));
router.get('/anki/changes', lazyloadRouteHandler('./routes/anki/changes'));
// ABC News
router.get('/abc/:site?', lazyloadRouteHandler('./routes/abc/index.js'));
router.get('/abc/:id?', lazyloadRouteHandler('./routes/abc'));
// 台湾中央通讯社
router.get('/cna/:id?', lazyloadRouteHandler('./routes/cna/index'));
File diff suppressed because it is too large Load Diff
+7272
View File
File diff suppressed because it is too large Load Diff
+40 -51
View File
@@ -1,30 +1,33 @@
const url = require('url');
const got = require('@/utils/got');
const cheerio = require('cheerio');
const { parseDate } = require('@/utils/parse-date');
const ids = require('./id');
const documentIds = require('./documentId');
module.exports = async (ctx) => {
ctx.params.site = ctx.params.site || '';
const id = ctx.params.id ?? 'justin';
const isChannel = ids.hasOwnProperty(id) ? !/topic\//.test(ids[id].link) : documentIds.hasOwnProperty(id) ? !/topic\//.test(documentIds[id].link) : isNaN(id) ? undefined : true;
if (isChannel === undefined) {
throw Error('Bad id. See <a href="https://docs.rsshub.app/traditional-media.html#abc-news-channel-topic">docs</a>');
}
const rootUrl = 'https://www.abc.net.au';
const currentUrl = `${rootUrl}/news/${ids.hasOwnProperty(id) ? ids[id].link : documentIds.hasOwnProperty(id) ? documentIds[id].link : id}`;
const apiUrl = `${rootUrl}/news-web/api/loader/${id === 'justin' ? 'justinstories?' : `${isChannel ? `channelrefetch` : `topicstories`}?name=PaginationArticles&documentId=${!isNaN(id) ? id : ids[id].documentId}&`}size=100`;
const rootUrl = 'https://mobile.abc.net.au';
const currentUrl = `${rootUrl}/news/${ctx.params.site}`;
const response = await got({
method: 'get',
url: currentUrl,
url: apiUrl,
});
const $ = cheerio.load(response.data);
$('._3zQMR, .ticker').remove();
const list = $('h3')
.slice(0, 10)
.map((_, item) => {
item = $(item);
return {
title: item.text(),
link: url.resolve(rootUrl, item.parents('a').attr('href') || item.find('a').attr('href')),
};
})
.get();
const list = response.data.collection.slice(0, ctx.query.limit ? parseInt(ctx.query.limit) : 50).map((item) => ({
title: item.title.children,
link: `${rootUrl}${item.link.to}`,
pubDate: parseDate(item.timestamp.dates.firstPublished.labelDate),
}));
const items = await Promise.all(
list.map((item) =>
@@ -33,44 +36,30 @@ module.exports = async (ctx) => {
method: 'get',
url: item.link,
});
const content = cheerio.load(detailResponse.data);
content('.modal-content').remove();
content('div[data-component="ZendeskForm"]').remove();
content('aside[data-component="RelatedCard"]').remove();
content('img').each(function () {
content(this).attr('src', content(this).attr('data-src'));
});
item.description = '';
let coverImage, contentBody;
if (content('div[data-component="FeatureMedia"]').html() !== null) {
coverImage = content('div[data-component="FeatureMedia"]').html();
} else {
coverImage = content('.view-hero-media').html();
}
if (content('#body').html() !== null) {
contentBody = content('#body').html();
} else {
contentBody = content('.article-text').html();
}
const cover = content('div[data-component="FeatureMedia"]');
const coverImage = cover.find('img');
coverImage.attr('src', coverImage.attr('data-src'));
const authorsArray = [];
const authorsMatch = detailResponse.data.match(/author":(.*),"dateModified/);
if (authorsMatch === null) {
authorsArray.push(content('meta[name="DCTERMS.contributor"]').attr('content'));
} else {
const authors = JSON.parse(authorsMatch[1]);
try {
for (const author of authors) {
authorsArray.push(author.name);
}
} catch (e) {
authorsArray.push(authors.name);
}
}
item.description += coverImage.toString() + cover.find('figcaption').toString();
item.description = coverImage + contentBody;
item.pubDate = new Date(content('meta[property="article:published_time"]').attr('content')).toUTCString();
item.author = authorsArray.join(', ');
content('.ZN39J')
.find('noscript')
.each(function () {
content(this).parent().parent().html(content(this).text());
});
item.description += content('.ZN39J').html();
item.category = content('meta[property="article:tag"]')
.toArray()
.map((tag) => content(tag).attr('content'));
return item;
})
@@ -78,7 +67,7 @@ module.exports = async (ctx) => {
);
ctx.state.data = {
title: $('title').text(),
title: `${ids.hasOwnProperty(id) ? ids[id].title : documentIds.hasOwnProperty(id) ? documentIds[id].title : id} - ABC`,
link: currentUrl,
item: items,
};