mirror/php-rss-bridge
1
0
Fork 0
mirror of https://github.com/RSS-Bridge/rss-bridge.git synced 2025-08-17 22:02:09 +02:00
Code Issues Releases Wiki Activity

Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802)

Browse Source 
* Debug mode improvements

 - Improve debug warning message
 - Restore error reporting in debug mode
 - Fix 'notice' messages for unset fields

* Add parsing utility functions

html.php
 - extractFromDelimiters
 - stripWithDelimiters
 - stripRecursiveHTMLSection
 - markdownToHtml (partial)

bridges
 - remove now-duplicate functions
 - call functions from html.php instead

* [Anidex] New bridge

Anime torrent tracker

* [Anime-Ultime] Restore thumbnail

* [CNET] Recreate bridge

Full rewrite as the previous one was broken

* [Dilbert] Minor URI fix

Use new self::URI property

* [EstCeQuonMetEnProd] Fix content extraction

Bridge was broken

* [Facebook] Fix "SpSonsSoriSsés" label

... which was taking space in item title

* [Futura-Sciences] Use HTTPS, More cleanup

Use HTTPS as FS now offer HTTPS
Clean additional useless HTML elements

* [GBATemp] Multiple fixes

- Fix categories: missing "break" statements
- Restore thumbnail as enclosure
- Fix date extraction
- Fix user blog post extraction
- Use getSimpleHTMLDOMCached

* [JapanExpo] Fix bridge, HTTPS, thumbnails

- Fix getSimpleHTMLDOMCached call
- Upgrade to HTTPS as JE now offers HTTPS
- Restore thumbnails as enclosures

* [LeMondeInformatique] Fix bridge, HTTPS

- Upgrade to HTTPS as LMI now offers HTTPS
- Restore thumbnails using small images
- Fix content extraction
- Fix text encoding issue

* [Nextgov] Fix content extraction

- Restore thumbnail and use small image
- Field extraction fixes

* [NextInpact] Add categories and filtering by type

- Offer all RSS feeds
- Allow filtering by article type
- Implement extraction for brief articles
- Remove article limit, many brief articles are publied all at once

* [NyaaTorrents] New bridge

Anime torrent tracker

* [Releases3DS] Cache content, restore thumbnail

- Use getSimpleHTMLDOMCached
- Restore thumbnail as enclosure

* [TheHackerNews] Fix bridge

 - Fix content extraction including article body
 - Restore thumbnail as enclosure

* [WeLiveSecurity] HTTPS, Fix content extraction

- Upgrade to HTTPS as WLS now offers HTTPS
- Fix content extraction including article body

* [WordPress] Reduce timeout, more content selectors

- Reduce timeout to use default one (1h)
- Add new content selector (articleBody)
- Find thumbnail and set as enclosure
- Fix <script> cleanup

* [YGGTorrent] Increase limit, use cache

- Increase item limit as uploads are very frequent
- Use getSimpleHTMLDOMCached

* [ZDNet] Rewrite with FeedExpander

- Upgrade to HTTPS as ZD now offers HTTPS
- Use FeedExpander for secondary fields
- Fix content extraction for article body

* [Main] Handle MIME type for enclosures

Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content).

One can force enclosure type using #.ext anchor (hacky, needs improving)

* [FeedExpander] Improve field extraction

- Add support for passing enclosures
- Improve author and uri extraction
- Fix 'notice' PHP error messages

* [Pull] Coding style fixes for #802

* [Pull] Implementing changes for #802

 - Fix coding style issues with str append
 - Remove useless CACHE_TIMEOUT
 - Use count() instead of $limit
 - Use defaultLinkTo() + handle strings
 - Use http_build_query()
 - Fix missing </em>
 - Remove error_reporting(0)
 - warning CSS (@LogMANOriginal)
 - Fix typo in FeedExpander comment

* [Main] More documentation for markdownToHtml

See #802 for more details
This commit is contained in:
ORelio
2018-09-09 21:20:13 +02:00
committed by Teromene
parent 123fce4394
commit de8cee6a1c
29 changed files with 1045 additions and 453 deletions
Download Patch File Download Diff File Expand all files Collapse all files

207
bridges/AnidexBridge.php Normal file
View File

@@ -0,0 +1,207 @@
<?php
class AnidexBridge extends BridgeAbstract {
const MAINTAINER = 'ORelio';
const NAME = 'Anidex';
const URI = 'https://anidex.info/';
const DESCRIPTION = 'Returns the newest torrents, with optional search criteria.';
const PARAMETERS = array(
array(
'id' => array(
'name' => 'Category',
'type' => 'list',
'values' => array(
'All categories' => '0',
'Anime' => '1,2,3',
'Anime - Sub' => '1',
'Anime - Raw' => '2',
'Anime - Dub' => '3',
'Live Action' => '4,5',
'Live Action - Sub' => '4',
'Live Action - Raw' => '5',
'Light Novel' => '6',
'Manga' => '7,8',
'Manga - Translated' => '7',
'Manga - Raw' => '8',
'Music' => '9,10,11',
'Music - Lossy' => '9',
'Music - Lossless' => '10',
'Music - Video' => '11',
'Games' => '12',
'Applications' => '13',
'Pictures' => '14',
'Adult Video' => '15',
'Other' => '16'
)
),
'lang_id' => array(
'name' => 'Language',
'type' => 'list',
'values' => array(
'All languages' => '0',
'English' => '1',
'Japanese' => '2',
'Polish' => '3',
'Serbo-Croatian' => '4',
'Dutch' => '5',
'Italian' => '6',
'Russian' => '7',
'German' => '8',
'Hungarian' => '9',
'French' => '10',
'Finnish' => '11',
'Vietnamese' => '12',
'Greek' => '13',
'Bulgarian' => '14',
'Spanish (Spain)' => '15',
'Portuguese (Brazil)' => '16',
'Portuguese (Portugal)' => '17',
'Swedish' => '18',
'Arabic' => '19',
'Danish' => '20',
'Chinese (Simplified)' => '21',
'Bengali' => '22',
'Romanian' => '23',
'Czech' => '24',
'Mongolian' => '25',
'Turkish' => '26',
'Indonesian' => '27',
'Korean' => '28',
'Spanish (LATAM)' => '29',
'Persian' => '30',
'Malaysian' => '31'
)
),
'group_id' => array(
'name' => 'Group ID',
'type' => 'number'
),
'r' => array(
'name' => 'Hide Remakes',
'type' => 'checkbox'
),
'b' => array(
'name' => 'Only Batches',
'type' => 'checkbox'
),
'a' => array(
'name' => 'Only Authorized',
'type' => 'checkbox'
),
'q' => array(
'name' => 'Keyword',
'description' => 'Keyword(s)',
'type' => 'text'
),
'h' => array(
'name' => 'Adult content',
'type' => 'list',
'values' => array(
'No filter' => '0',
'Hide +18' => '1',
'Only +18' => '2'
)
)
)
);
public function collectData() {
// Build Search URL from user-provided parameters
$search_url = self::URI . '?s=upload_timestamp&o=desc';
foreach (array('id', 'lang_id', 'group_id') as $param_name) {
$param = $this->getInput($param_name);
if (!empty($param) && intval($param) != 0 && ctype_digit(str_replace(',', '', $param))) {
$search_url .= '&' . $param_name . '=' . $param;
}
}
foreach (array('r', 'b', 'a') as $param_name) {
$param = $this->getInput($param_name);
if (!empty($param) && boolval($param)) {
$search_url .= '&' . $param_name . '=1';
}
}
$query = $this->getInput('q');
if (!empty($query)) {
$search_url .= '&q=' . urlencode($query);
}
$opt = array();
$h = $this->getInput('h');
if (!empty($h) && intval($h) != 0 && ctype_digit($h)) {
$opt[CURLOPT_COOKIE] = 'anidex_h_toggle=' . $h;
}
// Retrieve torrent listing from search results, which does not contain torrent description
$html = getSimpleHTMLDOM($search_url, array(), $opt)
or returnServerError('Could not request Anidex: ' . $search_url);
$links = $html->find('a');
$results = array();
foreach ($links as $link)
if (strpos($link->href, '/torrent/') === 0 && !in_array($link->href, $results))
$results[] = $link->href;
if (empty($results) && empty($this->getInput('q')))
returnServerError('No results from Anidex: '.$search_url);
//Process each item individually
foreach ($results as $element) {
//Limit total amount of requests
if(count($this->items) >= 20) {
break;
}
$torrent_id = str_replace('/torrent/', '', $element);
//Ignore entries without valid torrent ID
if ($torrent_id != 0 && ctype_digit($torrent_id)) {
//Retrieve data for this torrent ID
$item_uri = self::URI . 'torrent/'.$torrent_id;
//Retrieve full description from torrent page
if ($item_html = getSimpleHTMLDOMCached($item_uri)) {
//Retrieve data from page contents
$item_title = str_replace(' (Torrent) - AniDex ', '', $item_html->find('title', 0)->plaintext);
$item_desc = $item_html->find('div.panel-body', 0);
$item_author = trim($item_html->find('span.fa-user', 0)->parent()->plaintext);
$item_date = strtotime(trim($item_html->find('span.fa-clock', 0)->parent()->plaintext));
$item_image = $this->getURI() . 'images/user_logos/default.png';
//Check for description-less torrent andn optionally extract image
$desc_title_found = false;
foreach ($item_html->find('h3.panel-title') as $h3) {
if (strpos($h3, 'Description') !== false) {
$desc_title_found = true;
break;
}
}
if ($desc_title_found) {
//Retrieve image for thumbnail or generic logo fallback
foreach ($item_desc->find('img') as $img) {
if (strpos($img->src, 'prez') === false) {
$item_image = $img->src;
break;
}
}
$item_desc = trim($item_desc->innertext);
} else {
$item_desc = '<em>No description.</em>';
}
//Build and add final item
$item = array();
$item['uri'] = $item_uri;
$item['title'] = $item_title;
$item['author'] = $item_author;
$item['timestamp'] = $item_date;
$item['enclosures'] = array($item_image);
$item['content'] = $item_desc;
$this->items[] = $item;
}
}
$element = null;
}
$results = null;
}
}

20
bridges/AnimeUltimeBridge.php
View File

@@ -5,7 +5,7 @@ class AnimeUltimeBridge extends BridgeAbstract {
const NAME = 'Anime-Ultime';
const URI = 'http://www.anime-ultime.net/';
const CACHE_TIMEOUT = 10800; // 3h
const DESCRIPTION = 'Returns the 10 newest releases posted on Anime-Ultime';
const DESCRIPTION = 'Returns the newest releases posted on Anime-Ultime.';
const PARAMETERS = array( array(
'type' => array(
'name' => 'Type',
@@ -65,6 +65,13 @@ class AnimeUltimeBridge extends BridgeAbstract {
$item_link_element = $release->find('td', 0)->find('a', 0);
$item_uri = self::URI . $item_link_element->href;
$item_name = html_entity_decode($item_link_element->plaintext);
$item_image = self::URI . substr(
$item_link_element->onmouseover,
37,
strpos($item_link_element->onmouseover, ' ', 37) - 37
);
$item_episode = html_entity_decode(
str_pad(
$release->find('td', 1)->plaintext,
@@ -79,8 +86,7 @@ class AnimeUltimeBridge extends BridgeAbstract {
if(!empty($item_uri)) {
// Retrieve description from description page and
// convert relative image src info absolute image src
// Retrieve description from description page
$html_item = getContents($item_uri)
or returnServerError('Could not request Anime-Ultime: ' . $item_uri);
$item_description = substr(
@@ -91,10 +97,9 @@ class AnimeUltimeBridge extends BridgeAbstract {
0,
strpos($item_description, '<div id="table">')
);
$item_description = str_replace(
'src="images', 'src="' . self::URI . 'images',
$item_description
);
// Convert relative image src into absolute image src, remove line breaks
$item_description = defaultLinkTo($item_description, self::URI);
$item_description = str_replace("\r", '', $item_description);
$item_description = str_replace("\n", '', $item_description);
$item_description = utf8_encode($item_description);
@@ -105,6 +110,7 @@ class AnimeUltimeBridge extends BridgeAbstract {
$item['title'] = $item_name . ' ' . $item_type . ' ' . $item_episode;
$item['author'] = $item_fansub;
$item['timestamp'] = $item_date;
$item['enclosures'] = array($item_image);
$item['content'] = $item_description;
$this->items[] = $item;
$processedOK++;

109
bridges/CNETBridge.php Normal file
View File

@@ -0,0 +1,109 @@
<?php
class CNETBridge extends BridgeAbstract {
const MAINTAINER = 'ORelio';
const NAME = 'CNET News';
const URI = 'https://www.cnet.com/';
const CACHE_TIMEOUT = 3600; // 1h
const DESCRIPTION = 'Returns the newest articles.';
const PARAMETERS = array(
array(
'topic' => array(
'name' => 'Topic',
'type' => 'list',
'values' => array(
'All articles' => '',
'Apple' => 'apple',
'Google' => 'google',
'Microsoft' => 'tags-microsoft',
'Computers' => 'topics-computers',
'Mobile' => 'topics-mobile',
'Sci-Tech' => 'topics-sci-tech',
'Security' => 'topics-security',
'Internet' => 'topics-internet',
'Tech Industry' => 'topics-tech-industry'
)
)
)
);
private function cleanArticle($article_html) {
$offset_p = strpos($article_html, '<p>');
$offset_figure = strpos($article_html, '<figure');
$offset = ($offset_figure < $offset_p ? $offset_figure : $offset_p);
$article_html = substr($article_html, $offset);
$article_html = str_replace('href="/', 'href="' . self::URI, $article_html);
$article_html = str_replace(' height="0"', '', $article_html);
$article_html = str_replace('<noscript>', '', $article_html);
$article_html = str_replace('</noscript>', '', $article_html);
$article_html = StripWithDelimiters($article_html, '<a class="clickToEnlarge', '</a>');
$article_html = stripWithDelimiters($article_html, '<span class="nowPlaying', '</span>');
$article_html = stripWithDelimiters($article_html, '<span class="duration', '</span>');
$article_html = stripWithDelimiters($article_html, '<script', '</script>');
$article_html = stripWithDelimiters($article_html, '<svg', '</svg>');
return $article_html;
}
public function collectData() {
// Retrieve and check user input
$topic = str_replace('-', '/', $this->getInput('topic'));
if (!empty($topic) && (substr_count($topic, '/') > 1 || !ctype_alpha(str_replace('/', '', $topic))))
returnClientError('Invalid topic: ' . $topic);
// Retrieve webpage
$pageUrl = self::URI . (empty($topic) ? 'news/' : $topic.'/');
$html = getSimpleHTMLDOM($pageUrl)
or returnServerError('Could not request CNET: '.$pageUrl);
// Process articles
foreach($html->find('div.assetBody, div.riverPost') as $element) {
if(count($this->items) >= 10) {
break;
}
$article_title = trim($element->find('h2, h3', 0)->plaintext);
$article_uri = self::URI . substr($element->find('a', 0)->href, 1);
$article_thumbnail = $element->parent()->find('img[src]', 0)->src;
$article_timestamp = strtotime($element->find('time.assetTime, div.timeAgo', 0)->plaintext);
$article_author = trim($element->find('a[rel=author], a.name', 0)->plaintext);
$article_content = '<p><b>' . trim($element->find('p.dek', 0)->plaintext) . '</b></p>';
if (is_null($article_thumbnail))
$article_thumbnail = extractFromDelimiters($element->innertext, '<img src="', '"');
if (!empty($article_title) && !empty($article_uri) && strpos($article_uri, self::URI . 'news/') !== false) {
$article_html = getSimpleHTMLDOMCached($article_uri) or $article_html = null;
if (!is_null($article_html)) {
if (empty($article_thumbnail))
$article_thumbnail = $article_html->find('div.originalImage', 0);
if (empty($article_thumbnail))
$article_thumbnail = $article_html->find('span.imageContainer', 0);
if (is_object($article_thumbnail))
$article_thumbnail = $article_thumbnail->find('img', 0)->src;
$article_content .= trim(
$this->cleanArticle(
extractFromDelimiters(
$article_html, '<article', '<footer'
)
)
);
}
$item = array();
$item['uri'] = $article_uri;
$item['title'] = $article_title;
$item['author'] = $article_author;
$item['timestamp'] = $article_timestamp;
$item['enclosures'] = array($article_thumbnail);
$item['content'] = $article_content;
$this->items[] = $item;
}
}
}
}

4
bridges/DilbertBridge.php
View File

@@ -9,8 +9,8 @@ class DilbertBridge extends BridgeAbstract {
public function collectData(){
$html = getSimpleHTMLDOM($this->getURI())
or returnServerError('Could not request Dilbert: ' . $this->getURI());
$html = getSimpleHTMLDOM(self::URI)
or returnServerError('Could not request Dilbert: ' . self::URI);
foreach($html->find('section.comic-item') as $element) {

20
bridges/EstCeQuonMetEnProdBridge.php
View File

@@ -7,19 +7,9 @@ class EstCeQuonMetEnProdBridge extends BridgeAbstract {
const CACHE_TIMEOUT = 21600; // 6h
const DESCRIPTION = 'Should we put a website in production today? (French)';
public function collectData(){
function extractFromDelimiters($string, $start, $end){
if(strpos($string, $start) !== false) {
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
return $section_retrieved;
}
return false;
}
$html = getSimpleHTMLDOM($this->getURI())
or returnServerError('Could not request EstCeQuonMetEnProd: ' . $this->getURI());
public function collectData() {
$html = getSimpleHTMLDOM(self::URI)
or returnServerError('Could not request EstCeQuonMetEnProd: ' . self::URI);
$item = array();
$item['uri'] = $this->getURI() . '#' . date('Y-m-d');
@@ -28,8 +18,8 @@ class EstCeQuonMetEnProdBridge extends BridgeAbstract {
$item['timestamp'] = strtotime('today midnight');
$item['content'] = str_replace(
'src="/',
'src="' . $this->getURI(),
trim(extractFromDelimiters($html->outertext, '<body role="document">', '<br /><br />'))
'src="' . self::URI,
trim(extractFromDelimiters($html->outertext, '<body role="document">', '<div id="share'))
);
$this->items[] = $item;

22
bridges/FacebookBridge.php
View File

@@ -253,17 +253,6 @@ class FacebookBridge extends BridgeAbstract {
private function collectUserData(){
//Extract a string using start and end delimiters
function extractFromDelimiters($string, $start, $end){
if(strpos($string, $start) !== false) {
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
return $section_retrieved;
}
return false;
}
//Utility function for cleaning a Facebook link
$unescape_fb_link = function($matches){
if(is_array($matches) && count($matches) > 1) {
@@ -435,7 +424,7 @@ EOD;
$author = str_replace(' | Facebook', '', $html->find('title#pageTitle', 0)->innertext);
$profilePic = 'https://graph.facebook.com/'
. $this->getInput('u')
. '/picture?width=200&amp;height=200';
. '/picture?width=200&amp;height=200#.image';
$this->authorName = $author;
@@ -491,6 +480,12 @@ EOD;
'',
$content);
//Remove "SpSonsSoriSsés"
$content = preg_replace(
'/(?iU)<a [^>]+ href="#" role="link" [^>}]+>.+<\/a>/iU',
'',
$content);
//Remove html nodes, keep only img, links, basic formatting
$content = strip_tags($content, '<a><img><i><u><br><p>');
@@ -550,6 +545,9 @@ EOD;
$item['title'] = $title;
$item['author'] = $author;
$item['timestamp'] = $date;
if(strpos($item['content'], '<img') === false)
$item['enclosures'] = array($profilePic);
$this->items[] = $item;
}
}

55
bridges/FuturaSciencesBridge.php
View File

@@ -3,7 +3,7 @@ class FuturaSciencesBridge extends FeedExpander {
const MAINTAINER = 'ORelio';
const NAME = 'Futura-Sciences Bridge';
const URI = 'http://www.futura-sciences.com/';
const URI = 'https://www.futura-sciences.com/';
const DESCRIPTION = 'Returns the newest articles.';
const PARAMETERS = array( array(
@@ -90,42 +90,11 @@ class FuturaSciencesBridge extends FeedExpander {
or returnServerError('Could not request Futura-Sciences: ' . $item['uri']);
$item['content'] = $this->extractArticleContent($article);
$author = $this->extractAuthor($article);
$item['author'] = empty($author) ? $item['author'] : $author;
if (!empty($author))
$item['author'] = $author;
return $item;
}
private function stripWithDelimiters($string, $start, $end){
while(strpos($string, $start) !== false) {
$section_to_remove = substr($string, strpos($string, $start));
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
$string = str_replace($section_to_remove, '', $string);
} return $string;
}
private function stripRecursiveHTMLSection($string, $tag_name, $tag_start){
$open_tag = '<' . $tag_name;
$close_tag = '</' . $tag_name . '>';
$close_tag_length = strlen($close_tag);
if(strpos($tag_start, $open_tag) === 0) {
while(strpos($string, $tag_start) !== false) {
$max_recursion = 100;
$section_to_remove = null;
$section_start = strpos($string, $tag_start);
$search_offset = $section_start;
do {
$max_recursion--;
$section_end = strpos($string, $close_tag, $search_offset);
$search_offset = $section_end + $close_tag_length;
$section_to_remove = substr($string, $section_start, $section_end - $section_start + $close_tag_length);
$open_tag_count = substr_count($section_to_remove, $open_tag);
$close_tag_count = substr_count($section_to_remove, $close_tag);
} while ($open_tag_count > $close_tag_count && $max_recursion > 0);
$string = str_replace($section_to_remove, '', $string);
}
}
return $string;
}
private function extractArticleContent($article){
$contents = $article->find('section.article-text-classic', 0)->innertext;
$headline = trim($article->find('p.description', 0)->plaintext);
@@ -137,6 +106,7 @@ class FuturaSciencesBridge extends FeedExpander {
'<div class="sharebar2',
'<div class="diaporamafullscreen"',
'<div class="module social-button',
'<div class="module social-share',
'<div style="margin-bottom:10px;" class="noprint"',
'<div class="ficheprevnext',
'<div class="bar noprint',
@@ -148,16 +118,17 @@ class FuturaSciencesBridge extends FeedExpander {
'<div id="forumcomments',
'<div ng-if="active"'
) as $div_start) {
$contents = $this->stripRecursiveHTMLSection($contents, 'div', $div_start);
$contents = stripRecursiveHTMLSection($contents, 'div', $div_start);
}
$contents = $this->stripWithDelimiters($contents, '<hr ', '/>');
$contents = $this->stripWithDelimiters($contents, '<p class="content-date', '</p>');
$contents = $this->stripWithDelimiters($contents, '<h1 class="content-title', '</h1>');
$contents = $this->stripWithDelimiters($contents, 'fs:definition="', '"');
$contents = $this->stripWithDelimiters($contents, 'fs:xt:clicktype="', '"');
$contents = $this->stripWithDelimiters($contents, 'fs:xt:clickname="', '"');
$contents = $this->stripWithDelimiters($contents, '<script ', '</script>');
$contents = stripWithDelimiters($contents, '<hr ', '/>');
$contents = stripWithDelimiters($contents, '<p class="content-date', '</p>');
$contents = stripWithDelimiters($contents, '<h1 class="content-title', '</h1>');
$contents = stripWithDelimiters($contents, 'fs:definition="', '"');
$contents = stripWithDelimiters($contents, 'fs:xt:clicktype="', '"');
$contents = stripWithDelimiters($contents, 'fs:xt:clickname="', '"');
$contents = StripWithDelimiters($contents, '<section class="module-toretain module-propal-nl', '</section>');
$contents = stripWithDelimiters($contents, '<script ', '</script>');
return $headline . trim($contents);
}

108
bridges/GBAtempBridge.php
View File

@@ -20,50 +20,58 @@ class GBAtempBridge extends BridgeAbstract {
)
));
private function extractFromDelimiters($string, $start, $end){
if(strpos($string, $start) !== false) {
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
return $section_retrieved;
}
return false;
}
private function stripWithDelimiters($string, $start, $end){
while(strpos($string, $start) !== false) {
$section_to_remove = substr($string, strpos($string, $start));
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
$string = str_replace($section_to_remove, '', $string);
}
return $string;
}
private function buildItem($uri, $title, $author, $timestamp, $content){
private function buildItem($uri, $title, $author, $timestamp, $thumbnail, $content){
$item = array();
$item['uri'] = $uri;
$item['title'] = $title;
$item['author'] = $author;
$item['timestamp'] = $timestamp;
$item['content'] = $content;
if (!empty($thumbnail)) {
$item['enclosures'] = array($thumbnail);
}
return $item;
}
private function cleanupPostContent($content, $site_url){
$content = str_replace(':arrow:', '&#x27a4;', $content);
$content = str_replace('href="attachments/', 'href="'.$site_url.'attachments/', $content);
$content = $this->stripWithDelimiters($content, '<script', '</script>');
$content = stripWithDelimiters($content, '<script', '</script>');
return $content;
}
private function findItemDate($item){
$time = 0;
$dateField = $item->find('abbr.DateTime', 0);
if (is_object($dateField)) {
$time = intval(
extractFromDelimiters(
$dateField->outertext,
'data-time="',
'"'
)
);
} else {
$dateField = $item->find('span.DateTime', 0);
$time = DateTime::createFromFormat(
'M j, Y \a\t g:i A',
extractFromDelimiters(
$dateField->outertext,
'title="',
'"'
)
)->getTimestamp();
}
return $time;
}
private function fetchPostContent($uri, $site_url){
$html = getSimpleHTMLDOM($uri);
$html = getSimpleHTMLDOMCached($uri);
if(!$html) {
return 'Could not request GBAtemp ' . $uri;
return 'Could not request GBAtemp: ' . $uri;
}
$content = $html->find('div.messageContent', 0)->innertext;
$content = $html->find('div.messageContent, blockquote.baseHtml', 0)->innertext;
return $this->cleanupPostContent($content, $site_url);
}
@@ -76,70 +84,56 @@ class GBAtempBridge extends BridgeAbstract {
case 'N':
foreach($html->find('li[class=news_item full]') as $newsItem) {
$url = self::URI . $newsItem->find('a', 0)->href;
$time = intval(
$this->extractFromDelimiters(
$newsItem->find('abbr.DateTime', 0)->outertext,
'data-time="',
'"'
)
);
$img = $this->getURI() . $newsItem->find('img', 0)->src . '#.image';
$time = $this->findItemDate($newsItem);
$author = $newsItem->find('a.username', 0)->plaintext;
$title = $newsItem->find('a', 1)->plaintext;
$content = $this->fetchPostContent($url, self::URI);
$this->items[] = $this->buildItem($url, $title, $author, $time, $content);
$this->items[] = $this->buildItem($url, $title, $author, $time, $img, $content);
unset($newsItem); // Some items are heavy, freeing the item proactively helps saving memory
}
break;
case 'R':
foreach($html->find('li.portal_review') as $reviewItem) {
$url = self::URI . $reviewItem->find('a', 0)->href;
$img = $this->getURI() . extractFromDelimiters($reviewItem->find('a', 0)->style, 'image:url(', ')');
$title = $reviewItem->find('span.review_title', 0)->plaintext;
$content = getSimpleHTMLDOM($url)
or returnServerError('Could not request GBAtemp: ' . $uri);
$author = $content->find('a.username', 0)->plaintext;
$time = intval(
$this->extractFromDelimiters(
$content->find('abbr.DateTime', 0)->outertext,
'data-time="',
'"'
)
);
$time = $this->findItemDate($content);
$intro = '<p><b>' . ($content->find('div#review_intro', 0)->plaintext) . '</b></p>';
$review = $content->find('div#review_main', 0)->innertext;
$subheader = '<p><b>' . $content->find('div.review_subheader', 0)->plaintext . '</b></p>';
$procons = $content->find('table.review_procons', 0)->outertext;
$scores = $content->find('table.reviewscores', 0)->outertext;
$content = $this->cleanupPostContent($intro . $review . $subheader . $procons . $scores, self::URI);
$this->items[] = $this->buildItem($url, $title, $author, $time, $content);
$this->items[] = $this->buildItem($url, $title, $author, $time, $img, $content);
unset($reviewItem); // Free up memory
}
break;
case 'T':
foreach($html->find('li.portal-tutorial') as $tutorialItem) {
$url = self::URI . $tutorialItem->find('a', 0)->href;
$title = $tutorialItem->find('a', 0)->plaintext;
$time = intval(
$this->extractFromDelimiters(
$tutorialItem->find('abbr.DateTime', 0)->outertext,
'data-time="',
'"'
)
);
$time = $this->findItemDate($tutorialItem);
$author = $tutorialItem->find('a.username', 0)->plaintext;
$content = $this->fetchPostContent($url, self::URI);
$this->items[] = $this->buildItem($url, $title, $author, $time, $content);
$this->items[] = $this->buildItem($url, $title, $author, $time, null, $content);
unset($tutorialItem); // Free up memory
}
break;
case 'F':
foreach($html->find('li.rc_item') as $postItem) {
$url = self::URI . $postItem->find('a', 1)->href;
$title = $postItem->find('a', 1)->plaintext;
$time = intval(
$this->extractFromDelimiters(
$postItem->find('abbr.DateTime', 0)->outertext,
'data-time="',
'"'
)
);
$time = $this->findItemDate($postItem);
$author = $postItem->find('a.username', 0)->plaintext;
$content = $this->fetchPostContent($url, self::URI);
$this->items[] = $this->buildItem($url, $title, $author, $time, $content);
$this->items[] = $this->buildItem($url, $title, $author, $time, null, $content);
unset($postItem); // Free up memory
}
break;
}
}

8
bridges/JapanExpoBridge.php
View File

@@ -3,7 +3,7 @@ class JapanExpoBridge extends BridgeAbstract {
const MAINTAINER = 'Ginko';
const NAME = 'Japan Expo Actualités';
const URI = 'http://www.japan-expo-paris.com/fr/actualites';
const URI = 'https://www.japan-expo-paris.com/fr/actualites';
const CACHE_TIMEOUT = 14400; // 4h
const DESCRIPTION = 'Returns most recent entries from Japan Expo actualités.';
const PARAMETERS = array( array(
@@ -51,7 +51,7 @@ class JapanExpoBridge extends BridgeAbstract {
foreach($html->find('a._tile2') as $element) {
$url = $element->href;
$thumbnail = 'http://s.japan-expo.com/katana/images/JES049/paris.png';
$thumbnail = 'https://s.japan-expo.com/katana/images/JES049/paris.png';
preg_match('/url\(([^)]+)\)/', $element->find('img.rspvimgset', 0)->style, $img_search_result);
if(count($img_search_result) >= 2)
@@ -62,7 +62,8 @@ class JapanExpoBridge extends BridgeAbstract {
break;
}
$article_html = getSimpleHTMLDOMCached('Could not request JapanExpo: ' . $url);
$article_html = getSimpleHTMLDOMCached($url)
or returnServerError('Could not request JapanExpo: ' . $url);
$header = $article_html->find('header.pageHeadBox', 0);
$timestamp = strtotime($header->find('time', 0)->datetime);
$title_html = $header->find('div.section', 0)->next_sibling();
@@ -92,6 +93,7 @@ class JapanExpoBridge extends BridgeAbstract {
$item['uri'] = $url;
$item['title'] = $title;
$item['timestamp'] = $timestamp;
$item['enclosures'] = array($thumbnail);
$item['content'] = $content;
$this->items[] = $item;
$count++;

39
bridges/LeMondeInformatiqueBridge.php
View File

@@ -3,8 +3,7 @@ class LeMondeInformatiqueBridge extends FeedExpander {
const MAINTAINER = 'ORelio';
const NAME = 'Le Monde Informatique';
const URI = 'http://www.lemondeinformatique.fr/';
const CACHE_TIMEOUT = 1800; // 30min
const URI = 'https://www.lemondeinformatique.fr/';
const DESCRIPTION = 'Returns the newest articles.';
public function collectData(){
@@ -15,30 +14,26 @@ class LeMondeInformatiqueBridge extends FeedExpander {
$item = parent::parseItem($newsItem);
$article_html = getSimpleHTMLDOMCached($item['uri'])
or returnServerError('Could not request LeMondeInformatique: ' . $item['uri']);
$item['content'] = $this->cleanArticle($article_html->find('div#article', 0)->innertext);
$item['title'] = $article_html->find('h1.cleanprint-title', 0)->plaintext;
//Deduce thumbnail URL from article image URL
$item['enclosures'] = array(
str_replace(
'/grande/',
'/petite/',
$article_html->find('.article-image', 0)->find('img', 0)->src
)
);
//No response header sets the encoding, explicit conversion is needed or subsequent xml_encode() will fail
$item['content'] = utf8_encode($this->cleanArticle($article_html->find('div.col-primary', 0)->innertext));
$item['author'] = utf8_encode($article_html->find('div.author-infos', 0)->find('b', 0)->plaintext);
return $item;
}
private function stripCDATA($string){
$string = str_replace('<![CDATA[', '', $string);
$string = str_replace(']]>', '', $string);
return $string;
}
private function stripWithDelimiters($string, $start, $end){
while(strpos($string, $start) !== false) {
$section_to_remove = substr($string, strpos($string, $start));
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
$string = str_replace($section_to_remove, '', $string);
}
return $string;
}
private function cleanArticle($article_html){
$article_html = $this->stripWithDelimiters($article_html, '<script', '</script>');
$article_html = $this->stripWithDelimiters($article_html, '<h1 class="cleanprint-title"', '</h1>');
$article_html = stripWithDelimiters($article_html, '<script', '</script>');
$article_html = explode('<p class="contact-error', $article_html)[0] . '</div>';
return $article_html;
}
}

18
bridges/NeuviemeArtBridge.php
View File

@@ -6,16 +6,6 @@ class NeuviemeArtBridge extends FeedExpander {
const URI = 'http://www.9emeart.fr/';
const DESCRIPTION = 'Returns the newest articles.';
private function stripWithDelimiters($string, $start, $end){
while(strpos($string, $start) !== false) {
$section_to_remove = substr($string, strpos($string, $start));
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
$string = str_replace($section_to_remove, '', $string);
}
return $string;
}
protected function parseItem($item){
$item = parent::parseItem($item);
@@ -34,16 +24,16 @@ class NeuviemeArtBridge extends FeedExpander {
}
$article_content = '';
if($article_image) {
if ($article_image) {
$article_content = '<p><img src="' . $article_image . '" /></p>';
}
$article_content .= str_replace(
'src="/', 'src="' . self::URI,
$article_html->find('div.newsGenerique_con', 0)->innertext
);
$article_content = $this->stripWithDelimiters($article_content, '<script', '</script>');
$article_content = $this->stripWithDelimiters($article_content, '<style', '</style>');
$article_content = $this->stripWithDelimiters($article_content, '<link', '>');
$article_content = stripWithDelimiters($article_content, '<script', '</script>');
$article_content = stripWithDelimiters($article_content, '<style', '</style>');
$article_content = stripWithDelimiters($article_content, '<link', '>');
$item['content'] = $article_content;

104
bridges/NextInpactBridge.php
View File

@@ -6,29 +6,105 @@ class NextInpactBridge extends FeedExpander {
const URI = 'https://www.nextinpact.com/';
const DESCRIPTION = 'Returns the newest articles.';
const PARAMETERS = array( array(
'feed' => array(
'name' => 'Feed',
'type' => 'list',
'values' => array(
'Tous nos articles' => 'news',
'Nos contenus en accès libre' => 'acces-libre',
'Blog' => 'blog',
'Bons plans' => 'bonsplans'
)
),
'filter_premium' => array(
'name' => 'Premium',
'type' => 'list',
'values' => array(
'No filter' => '0',
'Hide Premium' => '1',
'Only Premium' => '2'
)
),
'filter_brief' => array(
'name' => 'Brief',
'type' => 'list',
'values' => array(
'No filter' => '0',
'Hide Brief' => '1',
'Only Brief' => '2'
)
)
));
public function collectData(){
$this->collectExpandableDatas(self::URI . 'rss/news.xml', 10);
$feed = $this->getInput('feed');
if (empty($feed))
$feed = 'news';
$this->collectExpandableDatas(self::URI . 'rss/' . $feed . '.xml');
}
protected function parseItem($newsItem){
$item = parent::parseItem($newsItem);
$item['content'] = $this->extractContent($item['uri']);
$item['content'] = $this->extractContent($item, $item['uri']);
if (is_null($item['content']))
return null; //Filtered article
return $item;
}
private function extractContent($url){
$html2 = getSimpleHTMLDOMCached($url);
$text = '<p><em>'
. $html2->find('span.sub_title', 0)->innertext
. '</em></p><p><img src="'
. $html2->find('div.container_main_image_article', 0)->find('img.dedicated', 0)->src
. '" alt="-" /></p><div>'
. $html2->find('div[itemprop=articleBody]', 0)->innertext
. '</div>';
private function extractContent($item, $url){
$html = getSimpleHTMLDOMCached($url);
if (!is_object($html))
return 'Failed to request NextInpact: ' . $url;
foreach(array(
'filter_premium' => 'h2.title_reserve_article',
'filter_brief' => 'div.brief-inner-content'
) as $param_name => $selector) {
$param_val = intval($this->getInput($param_name));
if ($param_val != 0) {
$element_present = is_object($html->find($selector, 0));
$element_wanted = ($param_val == 2);
if ($element_present != $element_wanted) {
return null; //Filter article
}
}
}
if (is_object($html->find('div[itemprop=articleBody], div.brief-inner-content', 0))) {
$subtitle = trim($html->find('span.sub_title, div.brief-head', 0));
if(is_object($subtitle) && $subtitle->plaintext !== $item['title']) {
$subtitle = '<p><em>' . $subtitle->plaintext . '</em></p>';
} else {
$subtitle = '';
}
$postimg = $html->find(
'div.container_main_image_article, div.image-brief-container, div.image-brief-side-container', 0
);
if(is_object($postimg)) {
$postimg = '<p><img src="'
. $postimg->find('img.dedicated', 0)->src
. '" alt="-" /></p>';
} else {
$postimg = '';
}
$text = $subtitle
. $postimg
. $html->find('div[itemprop=articleBody], div.brief-inner-content', 0)->outertext;
} else {
$text = $item['content']
. '<p><em>Failed retrieve full article content</em></p>';
}
$premium_article = $html->find('h2.title_reserve_article', 0);
if (is_object($premium_article)) {
$text .= '<p><em>' . $premium_article->innertext . '</em></p>';
}
$premium_article = $html2->find('h2.title_reserve_article', 0);
if (is_object($premium_article))
$text = $text . '<p><em>' . $premium_article->innertext . '</em></p>';
return $text;
}
}

42
bridges/NextgovBridge.php
View File

@@ -32,43 +32,39 @@ class NextgovBridge extends FeedExpander {
protected function parseItem($newsItem){
$item = parent::parseItem($newsItem);
$item['content'] = '';
$article_thumbnail = 'https://cdn.nextgov.com/nextgov/images/logo.png';
$item['content'] = '<p><b>' . $item['content'] . '</b></p>';
$namespaces = $newsItem->getNamespaces(true);
if(isset($namespaces['media'])) {
$media = $newsItem->children($namespaces['media']);
if(isset($media->content)) {
$attributes = $media->content->attributes();
$item['content'] = '<img src="' . $attributes['url'] . '">';
$item['content'] = '<p><img src="' . $attributes['url'] . '"></p>' . $item['content'];
$article_thumbnail = str_replace(
'large.jpg',
'small.jpg',
strval($attributes['url'])
);
}
}
$item['enclosures'] = array($article_thumbnail);
$item['content'] .= $this->extractContent($item['uri']);
return $item;
}
private function stripWithDelimiters($string, $start, $end){
while (strpos($string, $start) !== false) {
$section_to_remove = substr($string, strpos($string, $start));
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
$string = str_replace($section_to_remove, '', $string);
}
return $string;
}
private function extractContent($url){
$article = getSimpleHTMLDOMCached($url)
or returnServerError('Could not request Nextgov: ' . $url);
$article = getSimpleHTMLDOMCached($url);
$contents = $article->find('div.wysiwyg', 0)->innertext;
$contents = $this->stripWithDelimiters($contents, '<div class="ad-container">', '</div>');
$contents = $this->stripWithDelimiters($contents, '<div', '</div>'); //ad outer div
return $this->stripWithDelimiters($contents, '<script', '</script>');
$contents = ($article_thumbnail == '' ? '' : '<p><img src="' . $article_thumbnail . '" /></p>')
. '<p><b>'
. $article_subtitle
. '</b></p>'
. trim($contents);
if (!is_object($article))
return 'Could not request Nextgov: ' . $url;
$contents = $article->find('div.wysiwyg', 0);
$contents->find('svg.content-tombstone', 0)->outertext = '';
$contents = $contents->innertext;
$contents = stripWithDelimiters($contents, '<div class="ad-container">', '</div>');
$contents = stripWithDelimiters($contents, '<div', '</div>'); //ad outer div
return trim(stripWithDelimiters($contents, '<script', '</script>'));
}
}

127
bridges/NyaaTorrentsBridge.php Normal file
View File

@@ -0,0 +1,127 @@
<?php
class NyaaTorrentsBridge extends BridgeAbstract {
const MAINTAINER = 'ORelio';
const NAME = 'NyaaTorrents';
const URI = 'https://nyaa.si/';
const DESCRIPTION = 'Returns the newest torrents, with optional search criteria.';
const PARAMETERS = array(
array(
'f' => array(
'name' => 'Filter',
'type' => 'list',
'values' => array(
'No filter' => '0',
'No remakes' => '1',
'Trusted only' => '2'
)
),
'c' => array(
'name' => 'Category',
'type' => 'list',
'values' => array(
'All categories' => '0_0',
'Anime' => '1_0',
'Anime - AMV' => '1_1',
'Anime - English' => '1_2',
'Anime - Non-English' => '1_3',
'Anime - Raw' => '1_4',
'Audio' => '2_0',
'Audio - Lossless' => '2_1',
'Audio - Lossy' => '2_2',
'Literature' => '3_0',
'Literature - English' => '3_1',
'Literature - Non-English' => '3_2',
'Literature - Raw' => '3_3',
'Live Action' => '4_0',
'Live Action - English' => '4_1',
'Live Action - Idol/PV' => '4_2',
'Live Action - Non-English' => '4_3',
'Live Action - Raw' => '4_4',
'Pictures' => '5_0',
'Pictures - Graphics' => '5_1',
'Pictures - Photos' => '5_2',
'Software' => '6_0',
'Software - Apps' => '6_1',
'Software - Games' => '6_2',
)
),
'q' => array(
'name' => 'Keyword',
'description' => 'Keyword(s)',
'type' => 'text'
)
)
);
public function collectData() {
// Build Search URL from user-provided parameters
$search_url = self::URI . '?s=id&o=desc&'
. http_build_query(array(
'f' => $this->getInput('f'),
'c' => $this->getInput('c'),
'q' => $this->getInput('q')
));
// Retrieve torrent listing from search results, which does not contain torrent description
$html = getSimpleHTMLDOM($search_url)
or returnServerError('Could not request Nyaa: ' . $search_url);
$links = $html->find('a');
$results = array();
foreach ($links as $link)
if (strpos($link->href, '/view/') === 0 && !in_array($link->href, $results))
$results[] = $link->href;
if (empty($results) && empty($this->getInput('q')))
returnServerError('No results from Nyaa: ' . $url, 500);
//Process each item individually
foreach ($results as $element) {
//Limit total amount of requests
if(count($this->items) >= 20) {
break;
}
$torrent_id = str_replace('/view/', '', $element);
//Ignore entries without valid torrent ID
if ($torrent_id != 0 && ctype_digit($torrent_id)) {
//Retrieve data for this torrent ID
$item_uri = self::URI . 'view/' . $torrent_id;
//Retrieve full description from torrent page
if ($item_html = getSimpleHTMLDOMCached($item_uri)) {
//Retrieve data from page contents
$item_title = str_replace(' :: Nyaa', '', $item_html->find('title', 0)->plaintext);
$item_desc = str_get_html(markdownToHtml($item_html->find('#torrent-description', 0)->innertext));
$item_author = extractFromDelimiters($item_html->outertext, 'href="/user/', '"');
$item_date = intval(extractFromDelimiters($item_html->outertext, 'data-timestamp="', '"'));
//Retrieve image for thumbnail or generic logo fallback
$item_image = $this->getURI() . 'static/img/avatar/default.png';
foreach ($item_desc->find('img') as $img) {
if (strpos($img->src, 'prez') === false) {
$item_image = $img->src;
break;
}
}
//Build and add final item
$item = array();
$item['uri'] = $item_uri;
$item['title'] = $item_title;
$item['author'] = $item_author;
$item['timestamp'] = $item_date;
$item['enclosures'] = array($item_image);
$item['content'] = $item_desc;
$this->items[] = $item;
}
}
$element = null;
}
$results = null;
}
}

15
bridges/Releases3DSBridge.php
View File

@@ -9,16 +9,6 @@ class Releases3DSBridge extends BridgeAbstract {
public function collectData(){
function extractFromDelimiters($string, $start, $end){
if(strpos($string, $start) !== false) {
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
return $section_retrieved;
}
return false;
}
function typeToString($type){
switch($type) {
case 1: return '3DS Game';
@@ -76,8 +66,8 @@ class Releases3DSBridge extends BridgeAbstract {
$ignDate = time();
$ignCoverArt = '';
$ignSearchUrl = 'http://www.ign.com/search?q=' . urlencode($name);
if($ignResult = getSimpleHTMLDOM($ignSearchUrl)) {
$ignSearchUrl = 'https://www.ign.com/search?q=' . urlencode($name);
if($ignResult = getSimpleHTMLDOMCached($ignSearchUrl)) {
$ignCoverArt = $ignResult->find('div.search-item-media', 0)->find('img', 0)->src;
$ignDesc = $ignResult->find('div.search-item-description', 0)->plaintext;
$ignLink = $ignResult->find('div.search-item-sub-title', 0)->find('a', 1)->href;
@@ -127,6 +117,7 @@ class Releases3DSBridge extends BridgeAbstract {
$item['title'] = $name;
$item['author'] = $publisher;
$item['timestamp'] = $ignDate;
$item['enclosures'] = array($ignCoverArt);
$item['uri'] = empty($ignLink) ? $searchLinkDuckDuckGo : $ignLink;
$item['content'] = $ignDescription . $releaseDescription . $releaseSearchLinks;
$this->items[] = $item;

97
bridges/TheHackerNewsBridge.php
View File

@@ -8,67 +8,66 @@ class TheHackerNewsBridge extends BridgeAbstract {
public function collectData(){
function stripWithDelimiters($string, $start, $end){
while(strpos($string, $start) !== false) {
$section_to_remove = substr($string, strpos($string, $start));
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
$string = str_replace($section_to_remove, '', $string);
}
return $string;
}
function stripRecursiveHtmlSection($string, $tag_name, $tag_start){
$open_tag = '<' . $tag_name;
$close_tag = '</' . $tag_name . '>';
$close_tag_length = strlen($close_tag);
if(strpos($tag_start, $open_tag) === 0) {
while(strpos($string, $tag_start) !== false) {
$max_recursion = 100;
$section_to_remove = null;
$section_start = strpos($string, $tag_start);
$search_offset = $section_start;
do {
$max_recursion--;
$section_end = strpos($string, $close_tag, $search_offset);
$search_offset = $section_end + $close_tag_length;
$section_to_remove = substr(
$string,
$section_start,
$section_end - $section_start + $close_tag_length
);
$open_tag_count = substr_count($section_to_remove, $open_tag);
$close_tag_count = substr_count($section_to_remove, $close_tag);
} while($open_tag_count > $close_tag_count && $max_recursion > 0);
$string = str_replace($section_to_remove, '', $string);
}
}
return $string;
}
$html = getSimpleHTMLDOM($this->getURI())
or returnServerError('Could not request TheHackerNews: ' . $this->getURI());
$limit = 0;
foreach($html->find('article') as $element) {
foreach($html->find('div.body-post') as $element) {
if($limit < 5) {
$article_url = $element->find('a.entry-title', 0)->href;
$article_author = trim($element->find('span.vcard', 0)->plaintext);
$article_title = $element->find('a.entry-title', 0)->plaintext;
$article_timestamp = strtotime($element->find('span.updated', 0)->plaintext);
$article = getSimpleHTMLDOM($article_url)
or returnServerError('Could not request TheHackerNews: ' . $article_url);
$article_url = $element->find('a.story-link', 0)->href;
$article_author = trim($element->find('i.fa-user', 0)->parent()->plaintext);
$article_title = $element->find('h2.home-title', 0)->plaintext;
$contents = $article->find('div.articlebodyonly', 0)->innertext;
$contents = stripRecursiveHtmlSection($contents, 'div', '<div class=\'clear\'');
$contents = stripWithDelimiters($contents, '<script', '</script>');
//Date without time
$article_timestamp = strtotime(
extractFromDelimiters(
$element->find('i.fa-calendar', 0)->parent()->outertext,
'</i>',
'<span>'
)
);
//Article thumbnail in lazy-loading image
if (is_object($element->find('img[data-echo]', 0))) {
$article_thumbnail = array(
extractFromDelimiters(
$element->find('img[data-echo]', 0)->outertext,
"data-echo='",
"'"
)
);
} else {
$article_thumbnail = array();
}
if ($article = getSimpleHTMLDOMCached($article_url)) {
//Article body
$contents = $article->find('div.articlebody', 0)->innertext;
$contents = stripRecursiveHtmlSection($contents, 'div', '<div class="ad_');
$contents = stripWithDelimiters($contents, 'id="google_ads', '</iframe>');
$contents = stripWithDelimiters($contents, '<script', '</script>');
//Date with time
if (is_object($article->find('meta[itemprop=dateModified]', 0))) {
$article_timestamp = strtotime(
extractFromDelimiters(
$article->find('meta[itemprop=dateModified]', 0)->outertext,
"content='",
"'"
)
);
}
} else {
$contents = 'Could not request TheHackerNews: ' . $article_url;
}
$item = array();
$item['uri'] = $article_url;
$item['title'] = $article_title;
$item['author'] = $article_author;
$item['enclosures'] = $article_thumbnail;
$item['timestamp'] = $article_timestamp;
$item['content'] = trim($contents);
$this->items[] = $item;

29
bridges/WeLiveSecurityBridge.php
View File

@@ -3,37 +3,24 @@ class WeLiveSecurityBridge extends FeedExpander {
const MAINTAINER = 'ORelio';
const NAME = 'We Live Security';
const URI = 'http://www.welivesecurity.com/';
const URI = 'https://www.welivesecurity.com/';
const DESCRIPTION = 'Returns the newest articles.';
private function stripWithDelimiters($string, $start, $end){
while(strpos($string, $start) !== false) {
$section_to_remove = substr($string, strpos($string, $start));
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
$string = str_replace($section_to_remove, '', $string);
}
return $string;
}
protected function parseItem($item){
$item = parent::parseItem($item);
$article_html = getSimpleHTMLDOMCached($item['uri']);
if(!$article_html) {
$item['content'] .= '<p>Could not request ' . $this->getName() . ': ' . $item['uri'] . '</p>';
$item['content'] .= '<p><em>Could not request ' . $this->getName() . ': ' . $item['uri'] . '</em></p>';
return $item;
}
$article_content = $article_html->find('div.wlistingsingletext', 0)->innertext;
$article_content = $this->stripWithDelimiters($article_content, '<script', '</script>');
$article_content = '<p><b>'
. $item['content']
. '</b></p>'
. trim($article_content);
$item['content'] = $article_content;
$article_content = $article_html->find('div.formatted', 0)->innertext;
$article_content = stripWithDelimiters($article_content, '<script', '</script>');
$article_content = stripRecursiveHTMLSection($article_content, 'div', '<div class="comments');
$article_content = stripRecursiveHTMLSection($article_content, 'div', '<div class="similar-articles');
$article_content = stripRecursiveHTMLSection($article_content, 'span', '<span class="meta');
$item['content'] = trim($article_content);
return $item;
}

37
bridges/WordPressBridge.php
View File

@@ -3,8 +3,7 @@ class WordPressBridge extends FeedExpander {
const MAINTAINER = 'aledeg';
const NAME = 'Wordpress Bridge';
const URI = 'https://wordpress.org/';
const CACHE_TIMEOUT = 10800; // 3h
const DESCRIPTION = 'Returns the newest full posts of a Wordpress powered website';
const DESCRIPTION = 'Returns the newest full posts of a WordPress powered website';
const PARAMETERS = array( array(
'url' => array(
@@ -13,8 +12,8 @@ class WordPressBridge extends FeedExpander {
)
));
private function clearContent($content){
$content = preg_replace('/<script[^>]*>[^<]*<\/script>/', '', $content);
private function cleanContent($content){
$content = stripWithDelimiters($content, '<script', '</script>');
$content = preg_replace('/<div class="wpa".*/', '', $content);
$content = preg_replace('/<form.*\/form>/', '', $content);
return $content;
@@ -27,6 +26,10 @@ class WordPressBridge extends FeedExpander {
$article = null;
switch(true) {
case !is_null($article_html->find('[itemprop=articleBody]', 0)):
// highest priority content div
$article = $article_html->find('[itemprop=articleBody]', 0);
break;
case !is_null($article_html->find('article', 0)):
// most common content div
$article = $article_html->find('article', 0);
@@ -39,15 +42,37 @@ class WordPressBridge extends FeedExpander {
// another common content div
$article = $article_html->find('.post-content', 0);
break;
case !is_null($article_html->find('.post', 0)):
// for old WordPress themes without HTML5
$article = $article_html->find('.post', 0);
break;
}
foreach ($article->find('h1.entry-title') as $title)
if ($title->plaintext == $item['title'])
$title->outertext = '';
$article_image = $article_html->find('img.wp-post-image', 0);
if(!empty($item['content']) && (!is_object($article_image) || empty($article_image->src))) {
$article_image = str_get_html($item['content'])->find('img.wp-post-image', 0);
}
if(is_object($article_image) && !empty($article_image->src)) {
if(empty($article_image->getAttribute('data-lazy-src'))) {
$article_image = $article_image->src;
} else {
$article_image = $article_image->getAttribute('data-lazy-src');
}
$mime_type = getMimeType($article_image);
if (strpos($mime_type, 'image') === false)
$article_image .= '#.image'; // force image
if (empty($item['enclosures']))
$item['enclosures'] = array($article_image);
else
$item['enclosures'] = array_merge($item['enclosures'], $article_image);
}
if(!is_null($article)) {
$item['content'] = $this->clearContent($article->innertext);
$item['content'] = $this->cleanContent($article->innertext);
}
return $item;

6
bridges/YGGTorrentBridge.php
View File

@@ -110,8 +110,8 @@ class YGGTorrentBridge extends BridgeAbstract {
foreach($results->find('tr') as $row) {
$count++;
if($count == 1) continue;
if($count == 12) break;
if($count == 1) continue; // Skip table header
if($count == 22) break; // Stop processing after 21 items (20 + 1 table header)
$item = array();
$item['timestamp'] = $row->find('.hidden', 1)->plaintext;
$item['title'] = $row->find('a', 1)->plaintext;
@@ -135,7 +135,7 @@ class YGGTorrentBridge extends BridgeAbstract {
$url_full[5] = urlencode($url_full[5]);
$url_full[6] = urlencode($url_full[6]);
$url = implode('/', $url_full);
$page = getSimpleHTMLDOM($url) or returnServerError('Unable to query Yggtorrent page !');
$page = getSimpleHTMLDOMCached($url) or returnServerError('Unable to query Yggtorrent page !');
$author = $page->find('.informations', 0)->find('a', 4)->plaintext;
$content = $page->find('.default', 1);
return array('author' => $author, 'content' => $content);

157
bridges/ZDNetBridge.php
View File

@@ -1,9 +1,9 @@
<?php
class ZDNetBridge extends BridgeAbstract {
class ZDNetBridge extends FeedExpander {
const MAINTAINER = 'ORelio';
const NAME = 'ZDNet Bridge';
const URI = 'http://www.zdnet.com/';
const URI = 'https://www.zdnet.com/';
const DESCRIPTION = 'Technology News, Analysis, Comments and Product Reviews for IT Professionals.';
//http://www.zdnet.com/zdnet.opml
@@ -160,143 +160,42 @@ class ZDNetBridge extends BridgeAbstract {
));
public function collectData(){
function stripCdata($string){
$string = str_replace('<![CDATA[', '', $string);
$string = str_replace(']]>', '', $string);
return trim($string);
}
function extractFromDelimiters($string, $start, $end){
if(strpos($string, $start) !== false) {
$section_retrieved = substr($string, strpos($string, $start) + strlen($start));
$section_retrieved = substr($section_retrieved, 0, strpos($section_retrieved, $end));
return $section_retrieved;
}
return false;
}
function stripWithDelimiters($string, $start, $end){
while(strpos($string, $start) !== false) {
$section_to_remove = substr($string, strpos($string, $start));
$section_to_remove = substr($section_to_remove, 0, strpos($section_to_remove, $end) + strlen($end));
$string = str_replace($section_to_remove, '', $string);
}
return $string;
}
function stripRecursiveHtmlSection($string, $tag_name, $tag_start){
$open_tag = '<' . $tag_name;
$close_tag = '</' . $tag_name . '>';
$close_tag_length = strlen($close_tag);
if(strpos($tag_start, $open_tag) === 0) {
while(strpos($string, $tag_start) !== false) {
$max_recursion = 100;
$section_to_remove = null;
$section_start = strpos($string, $tag_start);
$search_offset = $section_start;
do {
$max_recursion--;
$section_end = strpos($string, $close_tag, $search_offset);
$search_offset = $section_end + $close_tag_length;
$section_to_remove = substr(
$string,
$section_start,
$section_end - $section_start + $close_tag_length
);
$open_tag_count = substr_count($section_to_remove, $open_tag);
$close_tag_count = substr_count($section_to_remove, $close_tag);
} while ($open_tag_count > $close_tag_count && $max_recursion > 0);
$string = str_replace($section_to_remove, '', $string);
}
}
return $string;
}
$baseUri = self::URI;
$baseUri = static::URI;
$feed = $this->getInput('feed');
if(strpos($feed, 'downloads!') !== false) {
$feed = str_replace('downloads!', '', $feed);
$baseUri = str_replace('www.', 'downloads.', $baseUri);
}
$url = $baseUri . trim($feed, '/') . '/rss.xml';
$html = getSimpleHTMLDOM($url)
or returnServerError('Could not request ZDNet: ' . $url);
$limit = 0;
$this->collectExpandableDatas($url);
}
foreach($html->find('item') as $element) {
if($limit < 10) {
$article_url = preg_replace(
'/([^#]+)#ftag=.*/',
'$1',
stripCdata(extractFromDelimiters($element->innertext, '<link>', '</link>'))
);
protected function parseItem($item){
$item = parent::parseItem($item);
$article_author = stripCdata(extractFromDelimiters($element->innertext, 'role="author">', '<'));
$article_title = stripCdata($element->find('title', 0)->plaintext);
$article_subtitle = stripCdata($element->find('description', 0)->plaintext);
$article_timestamp = strtotime(stripCdata($element->find('pubDate', 0)->plaintext));
$article = getSimpleHTMLDOM($article_url)
or returnServerError('Could not request ZDNet: ' . $article_url);
$article = getSimpleHTMLDOMCached($item['uri']);
if(!$article)
returnServerError('Could not request ZDNet: ' . $url);
if(!empty($article_author)) {
$author = $article_author;
} else {
$author = $article->find('meta[name=author]', 0);
if(is_object($author)) {
$author = $author->content;
} else {
$author = 'ZDNet';
}
}
$thumbnail = $article->find('meta[itemprop=image]', 0);
if(is_object($thumbnail)) {
$thumbnail = $thumbnail->content;
} else {
$thumbnail = '';
}
$contents = $article->find('article', 0)->innertext;
foreach(array(
'<div class="shareBar"',
'<div class="shortcodeGalleryWrapper"',
'<div class="relatedContent',
'<div class="downloadNow',
'<div data-shortcode',
'<div id="sharethrough',
'<div id="inpage-video'
) as $div_start) {
$contents = stripRecursiveHtmlSection($contents, 'div', $div_start);
}
$contents = stripWithDelimiters($contents, '<script', '</script>');
$contents = stripWithDelimiters($contents, '<meta itemprop="image"', '>');
$contents = trim(stripWithDelimiters($contents, '<section class="sharethrough-top', '</section>'));
$content_img = strpos($contents, '<img'); //Look for first image
if (($content_img !== false && $content_img < 512) || $thumbnail == '') {
$content_img = ''; //Image already present on article beginning or no thumbnail
} else {
$content_img = '<p><img src="'.$thumbnail.'" /></p>'; //Include thumbnail
}
$contents = $content_img
. '<p><b>'
. $article_subtitle
. '</b></p>'
. $contents;
$item = array();
$item['author'] = $author;
$item['uri'] = $article_url;
$item['title'] = $article_title;
$item['timestamp'] = $article_timestamp;
$item['content'] = $contents;
$this->items[] = $item;
$limit++;
}
$contents = $article->find('article', 0)->innertext;
foreach(array(
'<div class="shareBar"',
'<div class="shortcodeGalleryWrapper"',
'<div class="relatedContent',
'<div class="downloadNow',
'<div data-shortcode',
'<div id="sharethrough',
'<div id="inpage-video'
) as $div_start) {
$contents = stripRecursiveHtmlSection($contents, 'div', $div_start);
}
$contents = stripWithDelimiters($contents, '<script', '</script>');
$contents = stripWithDelimiters($contents, '<meta itemprop="image"', '>');
$contents = stripWithDelimiters($contents, '<svg class="svg-symbol', '</svg>');
$contents = trim(stripWithDelimiters($contents, '<section class="sharethrough-top', '</section>'));
$item['content'] = $contents;
return $item;
}
}