1
0
mirror of https://github.com/RSS-Bridge/rss-bridge.git synced 2025-09-18 11:51:33 +02:00

Compare commits

..

31 Commits

Author SHA1 Message Date
logmanoriginal
de7622ebbf version: Bump to 2018-08-07 2018-08-07 18:37:38 +02:00
logmanoriginal
09c9d015b4 [ForGifsBridge] Add new bridge 2018-08-04 23:42:58 +02:00
logmanoriginal
3a496e3b18 [FilterBridge] Add option to build title from content
Adds a new option '&title_from_content=on' to build the title for feed
items from the feeds content. The title is generated from the first
whitespace after 50 characters of the content or the entire content if
the total size is lower than 50 characters.

References #587
2018-08-04 20:46:59 +02:00
Eugene Molotov
df58f5bbdb [core] Add urljoin (#756)
Adds php-urljoin from https://github.com/fluffy-critter/php-urljoin to replace the custom implementation of 'defaultLinkTo'
2018-08-02 06:31:56 +02:00
logmanoriginal
9d0452d11b [.travis] Use composer for HHVM
This fixes the HHVM build failing because pear doesn't exist in HHVM.
2018-08-01 19:37:10 +02:00
sublimz
f92ac49947 [LeBonCoinBridge] Add cities support (#751) 2018-08-01 17:25:18 +02:00
Benasse
a574fa15ac [YGGTorrentBridge] Order search result by publish date (#762) 2018-07-31 21:46:10 +02:00
Nemo
8f9a385b4d [AmazonPriceTrackerBridge] Improve Amazon scraper logic (#761)
- Now works on all websites, and even with products
  with multiple prices
- Closes #750
2018-07-31 21:44:37 +02:00
logmanoriginal
53bdfa3bf0 [GooglePlusPostBridge] Skip posts without message 2018-07-31 19:15:09 +02:00
logmanoriginal
53278b2eed [GooglePlusPostBridge] Add option to include image in content
References #600
2018-07-31 19:09:12 +02:00
logmanoriginal
5f3c55b808 [GooglePlusPostBridge] General cleanup 2018-07-31 18:55:35 +02:00
logmanoriginal
fb79a67370 [GooglePlusPostBridge] Normalize static::URI usage
This commit fixes a few things related to static::URI

1) Remove trailing slash from the URI to simplify using 'defaultLinkTo'
2) Use static::URI instead of self::URI for consistency
3) Remove custom implementation of 'defaultLinkTo'
2018-07-31 18:29:14 +02:00
logmanoriginal
3c4e12ceba [GooglePlusPostBridge] Add images to enclosures
Images are collected for each post and added to enclosures. Images or
animtions from lh3.googleusercontent.com are specifically handled in
order to return the animated version of the gif and the original sized
image (this is normally taken care of by JS in the browser).
2018-07-31 18:18:22 +02:00
logmanoriginal
0d1923c52f [GitHubGistBridge] Add new bridge
Adds a new bridge for https://gist.github.com

The bridge generates feeds for comments on a particular gist based on
the gist ID or full URI. For better readability the general behavior
of code sections is manually restored with the original CSS styles
from GitHub.
2018-07-29 16:31:47 +02:00
logmanoriginal
ce896b4247 [SkimfeedBridge] Add new bridge
New bridge for Skimfeed: https://skimfeed.com

Generates feeds for all features of Skimfeed:

- News (the ones displayed on the front page)
- Hot topics ("What's Hot" section on the front page)
- Tech news (preconfigured feeds in the menu bar)
- Custom feeds (using the configuration system of Skimfeed), see
https://skimfeed.com/custom.php

The number of items returned by the bridge can be limited for all
categories ('&limit=...'). This parameter is optional, all categories
are unlimited by default!

Authors are added with HTML anchors in order to allow quick navigation
to source channels.

The bridge ships with developer tools to auto-generate lists in the
future (especially useful for 'Tech news'!)

References #748
2018-07-27 23:18:32 +02:00
sysadminstory
a4b2d88dbe [DealabsBridge] Follow website change (#758) 2018-07-25 20:02:31 +02:00
logmanoriginal
65ec04ea98 [contents] Remove superfluous debug log from getContents
References #757
2018-07-25 19:56:46 +02:00
logmanoriginal
afb4de318b [FlickrBridge] Fix missing scheme for image URLs
References #754
2018-07-23 20:14:46 +02:00
Eugene Molotov
43bb17f995 [VkBridge] Converting hashtags to categories (#755)
* [VkBridge] Converting hashtags to categories
2018-07-22 16:43:00 +02:00
logmanoriginal
bae7a5879f [FlickrBridge] Fixed broken bridge
Following changes in the JSON data and selecting images for the
content (320x240 or bigger) and enclosure (largest version). All of
the data is now extracted from the JSON data instead of parsing the
DOM.

References #754
2018-07-22 14:06:04 +02:00
LogMANOriginal
bd760cbcee [README] Add docker build status 2018-07-21 21:59:48 +02:00
LogMANOriginal
cd20b4476f [README] Add label for latest release 2018-07-21 21:54:46 +02:00
LogMANOriginal
d83f2f285b Separate index and bridge card generating code into a separate classes (#734)
[html] Generate index and bridge cards using separate clases

Move HTML generating code from 'index.php' to 'Index.php', separating components into static functions.

Move HTML generation code for bridge cards from 'html.php' to 'BridgeCard.php', separating components into static functions.
2018-07-21 18:15:07 +02:00
logmanoriginal
15e6d77569 [FierPandaBridge] Fix bridge
This bridge now returns all articles from the front page, following
layout changes in the past.

References #679
2018-07-21 18:07:03 +02:00
logmanoriginal
f97d2ef254 [Torrent9Bridge] Remove bridge
The site moved from www.torrent9.pe to www.t9.pe and is now protected
by Cloudflare challenges, making it inaccessible to RSS-Bridge.
2018-07-21 17:45:22 +02:00
logmanoriginal
91ae2a23d7 [CpasbienBridge] Remove bridge
Removing this bridge for two reasons:

1) The service moved from www.cpasbien.cm to www.torrents9.blue,
changing the layout in the process (incompatible).

2) The new site is permanently protected by Cloudflare IUAM, making
it inaccessible by RSS-Bridge.

While it would certainly be possible to rewrite the bridge to work
with the new layout, the site is still inaccessible.

References #605
2018-07-21 17:43:29 +02:00
logmanoriginal
066ef1d7db [contents] Add Cloudflare challenge detection
Adds detection for servers responding with Cloudflare challenges,
throwing a server error if detected:

"The server responded with a Cloudflare challenge, which is not
supported by RSS-Bridge! If this error persists longer than a week,
please consider opening an issue on GitHub!"

This is supposed to support maintainers to identify broken bridges
for sites with Cloudflare enabled permanently. It doesn't circumvent
the protection in any form or shape!

The Cloudflare challenge is detected by analyzing the last response
header received from the server. If the HTTP Code is not 200 (OK)
and the server name contains 'cloudflare' ('Server: cloudflare'),
RSS-Bridge assumes the server responded with a challenge.

The header parsing is based on https://stackoverflow.com/a/18682872
2018-07-21 17:43:29 +02:00
LogMANOriginal
4facbf32e3 [InstructableBridge] Add new bridge (#724)
This commit adds a new bridge for http://www.instructables.com. This bridge
currently supports fetching content by category (all categories available 200+),
using available filters (featured, recent, popular, views, contest winners).
2018-07-21 15:25:13 +02:00
logmanoriginal
6bd76af326 [YoutubeBridge] Add duration limits for all modes
Adds duration limits (minimum duration, maximum duration) for all
modes (user/id/playlist/search). Duration limits are optional, so
existing subscriptions don't break.

The limits are specified by two separate parameters, each of which
is optional:

- `&duration_min=` (minimum duration in minutes, default: -1)
- `&duration_max=` (maximum duration in minutes, default: INF)

If duration limits are specified in either user, id or playlist mode,
the bridge defaults to fetching data from HTML intead of XML feeds,
which requires more bandwidth and takes longer, because each video is
loaded individually!

References #670
2018-07-21 14:33:07 +02:00
logmanoriginal
caa622ffec [search] Support searching by URI
Adds matching for URIs to the search bar, using the format
<scheme>://<host>/<path>

Searching by URI scheme is also supported:

"http://"  (returns all bridges with 'http'  scheme)
"https://" (returns all bridges with 'https' scheme)

The following examples are equivalent and will return both of the
Facebook bridges (FacebookBridge and FB2Bridge):

"https://www.facebook.com/facebook"
"https://www.facebook.com/facebook?..."
"https://www.facebook.com"
"http://www.facebook.com"
"https://facebook.com"
"http://facebook.com"
"facebook.com"
"facebook"

Notice: When the URI scheme is omitted, the search algorithm falls back
to regex matching. Searching for "www.facebook.com" doesn't work, as it
is missing the schema and doesn't match via regex!

Omitting the 'www.', however, does work. This was a design decision for
some bridges specify their URI with and others without 'www.'

A search term can still be specified in the browser URL using parameter
'q' => '?q=searchterm'.

References #743
2018-07-20 22:44:13 +02:00
teromene
c4d489f018 Add URI to ElloBridge elements. 2018-07-19 17:07:54 +02:00
28 changed files with 2463 additions and 716 deletions

View File

@@ -3,12 +3,20 @@ sudo: false
language: php
install:
- pear channel-update pear.php.net
- pear install PHP_CodeSniffer
- if [[ $TRAVIS_PHP_VERSION == "hhvm" ]]; then
composer global require squizlabs/PHP_CodeSniffer;
else
pear channel-update pear.php.net;
pear install PHP_CodeSniffer;
fi
script:
- phpenv rehash
- phpcs . --standard=phpcs.xml --warning-severity=0 --extensions=php -p
- if [[ $TRAVIS_PHP_VERSION == "hhvm" ]]; then
/home/travis/.composer/vendor/bin/phpcs . --standard=phpcs.xml --warning-severity=0 --extensions=php -p;
else
phpcs . --standard=phpcs.xml --warning-severity=0 --extensions=php -p;
fi
matrix:
fast_finish: true

View File

@@ -1,6 +1,6 @@
rss-bridge
===
[![LICENSE](https://img.shields.io/badge/license-UNLICENSE-blue.svg)](UNLICENSE) [![Build Status](https://travis-ci.org/RSS-Bridge/rss-bridge.svg?branch=master)](https://travis-ci.org/RSS-Bridge/rss-bridge)
[![LICENSE](https://img.shields.io/badge/license-UNLICENSE-blue.svg)](UNLICENSE) [![GitHub release](https://img.shields.io/github/release/rss-bridge/rss-bridge.svg)](https://github.com/rss-bridge/rss-bridge/releases/latest) [![Build Status](https://travis-ci.org/RSS-Bridge/rss-bridge.svg?branch=master)](https://travis-ci.org/RSS-Bridge/rss-bridge) [![Docker Build Status](https://img.shields.io/docker/build/rssbridge/rss-bridge.svg)](https://hub.docker.com/r/rssbridge/rss-bridge/)
rss-bridge is a PHP project capable of generating ATOM feeds for websites which don't have one.

View File

@@ -92,6 +92,14 @@ class AmazonPriceTrackerBridge extends BridgeAbstract {
}
}
private function parseDynamicImage($attribute) {
$json = json_decode(html_entity_decode($attribute), true);
if ($json and count($json) > 0) {
return array_keys($json)[0];
}
}
/**
* Returns a generated image tag for the product
*/
@@ -99,11 +107,15 @@ class AmazonPriceTrackerBridge extends BridgeAbstract {
$imageSrc = $html->find('#main-image-container img', 0);
if ($imageSrc) {
$imageSrc = $imageSrc ? $imageSrc->getAttribute('data-old-hires') : '';
return <<<EOT
<img width="300" style="max-width:300;max-height:300" src="$imageSrc" alt="{$this->title}" />
EOT;
$hiresImage = $imageSrc->getAttribute('data-old-hires');
$dynamicImageAttribute = $imageSrc->getAttribute('data-a-dynamic-image');
$image = $hiresImage ?: $this->parseDynamicImage($dynamicImageAttribute);
}
$image = $image ?: 'https://placekitten.com/200/300';
return <<<EOT
<img width="300" style="max-width:300;max-height:300" src="$image" alt="{$this->title}" />
EOT;
}
/**
@@ -116,6 +128,39 @@ EOT;
return getSimpleHTMLDOM($uri) ?: returnServerError('Could not request Amazon.');
}
private function scrapePriceFromMetrics($html) {
$asinData = $html->find('#cerberus-data-metrics', 0);
// <div id="cerberus-data-metrics" style="display: none;"
// data-asin="B00WTHJ5SU" data-asin-price="14.99" data-asin-shipping="0"
// data-asin-currency-code="USD" data-substitute-count="-1" ... />
if ($asinData) {
return [
'price' => $asinData->getAttribute('data-asin-price'),
'currency' => $asinData->getAttribute('data-asin-currency-code'),
'shipping' => $asinData->getAttribute('data-asin-shipping')
];
}
return false;
}
private function scrapePriceGeneric($html) {
$priceDiv = $html->find('span.offer-price', 0) ?: $html->find('.a-color-price', 0);
preg_match('/^\s*([A-Z]{3}|£|\$)\s?([\d.,]+)\s*$/', $priceDiv->plaintext, $matches);
if (count($matches) === 3) {
return [
'price' => $matches[2],
'currency' => $matches[1],
'shipping' => '0'
];
}
return false;
}
/**
* Scrape method for Amazon product page
* @return [type] [description]
@@ -125,23 +170,16 @@ EOT;
$this->title = $this->getTitle($html);
$imageTag = $this->getImage($html);
$asinData = $html->find('#cerberus-data-metrics', 0);
// <div id="cerberus-data-metrics" style="display: none;"
// data-asin="B00WTHJ5SU" data-asin-price="14.99" data-asin-shipping="0"
// data-asin-currency-code="USD" data-substitute-count="-1" ... />
$currency = $asinData->getAttribute('data-asin-currency-code');
$shipping = $asinData->getAttribute('data-asin-shipping');
$price = $asinData->getAttribute('data-asin-price');
$data = $this->scrapePriceFromMetrics($html) ?: $this->scrapePriceGeneric($html);
$item = array(
'title' => $this->title,
'uri' => $this->getURI(),
'content' => "$imageTag<br/>Price: $price $currency",
'content' => "$imageTag<br/>Price: {$data['price']} {$data['currency']}",
);
if ($shipping !== '0') {
$item['content'] .= "<br>Shipping: $shipping $currency</br>";
if ($data['shipping'] !== '0') {
$item['content'] .= "<br>Shipping: {$data['shipping']} {$data['currency']}</br>";
}
$this->items[] = $item;

View File

@@ -1,74 +0,0 @@
<?php
class CpasbienBridge extends BridgeAbstract {
const MAINTAINER = 'lagaisse';
const NAME = 'Cpasbien Bridge';
const URI = 'http://www.cpasbien.cm';
const CACHE_TIMEOUT = 86400; // 24h
const DESCRIPTION = 'Returns latest torrents from a request query';
const PARAMETERS = array( array(
'q' => array(
'name' => 'Search',
'required' => true,
'title' => 'Type your search'
)
));
public function collectData(){
$request = str_replace(' ', '-', trim($this->getInput('q')));
$html = getSimpleHTMLDOM(self::URI . '/recherche/' . urlencode($request) . '.html')
or returnServerError('No results for this query.');
foreach($html->find('#gauche', 0)->find('div') as $episode) {
if($episode->getAttribute('class') == 'ligne0'
|| $episode->getAttribute('class') == 'ligne1') {
$urlepisode = $episode->find('a', 0)->getAttribute('href');
$htmlepisode = getSimpleHTMLDOMCached($urlepisode, 86400 * 366 * 30);
$item = array();
$item['author'] = $episode->find('a', 0)->text();
$item['title'] = $episode->find('a', 0)->text();
$item['pubdate'] = $this->getCachedDate($urlepisode);
$textefiche = $htmlepisode->find('#textefiche', 0)->find('p', 1);
if(isset($textefiche)) {
$item['content'] = $textefiche->text();
} else {
$p = $htmlepisode->find('#textefiche', 0)->find('p');
if(!empty($p)) {
$item['content'] = $htmlepisode->find('#textefiche', 0)->find('p', 0)->text();
}
}
$item['id'] = $episode->find('a', 0)->getAttribute('href');
$item['uri'] = self::URI . $htmlepisode->find('#telecharger', 0)->getAttribute('href');
$this->items[] = $item;
}
}
}
public function getName(){
if(!is_null($this->getInput('q'))) {
return $this->getInput('q') . ' : ' . self::NAME;
}
return parent::getName();
}
private function getCachedDate($url){
debugMessage('getting pubdate from url ' . $url . '');
// Initialize cache
$cache = Cache::create('FileCache');
$cache->setPath(CACHE_DIR . '/pages');
$params = [$url];
$cache->setParameters($params);
// Get cachefile timestamp
$time = $cache->getTime();
return ($time !== false ? $time : time());
}
}

View File

@@ -241,9 +241,7 @@ class PepperBridgeAbstract extends BridgeAbstract {
' ', /* Notice this is a space! */
array(
'cept-description-container',
'overflow--wrap-break',
'size--all-s',
'size--fromW3-m'
'overflow--wrap-break'
)
);

View File

@@ -45,9 +45,10 @@ class ElloBridge extends BridgeAbstract {
$item = array();
$item['author'] = $this->getUsername($post, $postData);
$item['timestamp'] = strtotime($post->created_at);
$item['title'] = $this->findText($post->summary);
$item['title'] = strip_tags($this->findText($post->summary));
$item['content'] = $this->getPostContent($post->body);
$item['enclosures'] = $this->getEnclosures($post, $postData);
$item['uri'] = self::URI . $item['author'] . '/post/' . $post->token;
$content = $post->body;
$this->items[] = $item;

View File

@@ -8,17 +8,22 @@ class FierPandaBridge extends BridgeAbstract {
const DESCRIPTION = 'Returns latest articles from Fier Panda.';
public function collectData(){
$html = getSimpleHTMLDOM(self::URI)
or returnServerError('Could not request Fier Panda.');
foreach($html->find('div.container-content article') as $element) {
defaultLinkTo($html, static::URI);
foreach($html->find('article') as $article) {
$item = array();
$item['uri'] = $this->getURI() . $element->find('a', 0)->href;
$item['title'] = trim($element->find('h1 a', 0)->innertext);
// Remove the link at the end of the article
$element->find('p a', 0)->outertext = '';
$item['content'] = $element->find('p', 0)->innertext;
$item['uri'] = $article->find('a', 0)->href;
$item['title'] = $article->find('a', 0)->title;
$this->items[] = $item;
}
}
}

View File

@@ -26,11 +26,34 @@ class FilterBridge extends FeedExpander {
),
'defaultValue' => 'permit',
),
'title_from_content' => array(
'name' => 'Generate title from content',
'type' => 'checkbox',
'required' => false,
)
));
protected function parseItem($newItem){
$item = parent::parseItem($newItem);
if($this->getInput('title_from_content') && array_key_exists('content', $item)) {
$content = str_get_html($item['content']);
$pos = strpos($item['content'], ' ', 50);
$item['title'] = substr(
$content->plaintext,
0,
$pos
);
if(strlen($content->plaintext) >= $pos) {
$item['title'] .= '...';
}
}
switch(true) {
case $this->getFilterType() === 'permit':
if (preg_match($this->getFilter(), $item['title'])) {

View File

@@ -30,30 +30,76 @@ class FlickrBridge extends BridgeAbstract {
'title' => 'Insert username (as shown in the address bar)',
'exampleValue' => 'flickr'
)
),
)
);
public function collectData(){
switch($this->queriedContext) {
case 'Explore':
$key = 'photos';
$filter = 'photo-lite-models';
$html = getSimpleHTMLDOM(self::URI . 'explore')
or returnServerError('Could not request Flickr.');
break;
case 'By keyword':
$key = 'photos';
$filter = 'photo-lite-models';
$html = getSimpleHTMLDOM(self::URI . 'search/?q=' . urlencode($this->getInput('q')) . '&s=rec')
or returnServerError('No results for this query.');
break;
case 'By username':
$key = 'photoPageList';
$filter = 'photo-models';
$html = getSimpleHTMLDOM(self::URI . 'photos/' . urlencode($this->getInput('u')))
or returnServerError('Requested username can\'t be found.');
break;
default:
returnClientError('Invalid context: ' . $this->queriedContext);
}
$model_json = $this->extractJsonModel($html);
$photo_models = $this->getPhotoModels($model_json, $filter);
foreach($photo_models as $model) {
$item = array();
/* Author name depends on scope. On a keyword search the
* author is part of the picture data. On a username search
* the author is part of the owner data.
*/
if(array_key_exists('username', $model)) {
$item['author'] = $model['username'];
} elseif (array_key_exists('owner', reset($model_json)[0])) {
$item['author'] = reset($model_json)[0]['owner']['username'];
}
$item['title'] = (array_key_exists('title', $model) ? $model['title'] : 'Untitled');
$item['uri'] = self::URI . 'photo.gne?id=' . $model['id'];
$description = (array_key_exists('description', $model) ? $model['description'] : '');
$item['content'] = '<a href="'
. $item['uri']
. '"><img src="'
. $this->extractContentImage($model)
. '" style="max-width: 640px; max-height: 480px;"/></a><br><p>'
. $description
. '</p>';
$item['enclosures'] = $this->extractEnclosures($model);
$this->items[] = $item;
}
}
private function extractJsonModel($html) {
// Find SCRIPT containing JSON data
$model = $html->find('.modelExport', 0);
$model_text = $model->innertext;
@@ -62,59 +108,79 @@ class FlickrBridge extends BridgeAbstract {
$start = strpos($model_text, 'modelExport:') + strlen('modelExport:');
$end = strpos($model_text, 'auth:') - strlen('auth:');
// Dissect JSON data and remove trailing comma
// Extract JSON data, remove trailing comma
$model_text = trim(substr($model_text, $start, $end - $start));
$model_text = substr($model_text, 0, strlen($model_text) - 1);
$model_json = json_decode($model_text, true);
return json_decode($model_text, true);
foreach($html->find('.photo-list-photo-view') as $element) {
// Get the styles
$style = explode(';', $element->style);
// Get the background-image style
$backgroundImage = explode(':', end($style));
// URI type : url(//cX.staticflickr.com/X/XXXXX/XXXXXXXXX.jpg)
$imageURI = trim(str_replace(['url(', ')'], '', end($backgroundImage)));
// Get the image ID
$imageURIs = explode('_', basename($imageURI));
$imageID = reset($imageURIs);
// Use JSON data to build items
foreach(reset($model_json)[0][$key]['_data'] as $element) {
if($element['id'] === $imageID) {
$item = array();
/* Author name depends on scope. On a keyword search the
* author is part of the picture data. On a username search
* the author is part of the owner data.
*/
if(array_key_exists('username', $element)) {
$item['author'] = $element['username'];
} elseif (array_key_exists('owner', reset($model_json)[0])) {
$item['author'] = reset($model_json)[0]['owner']['username'];
}
$item['title'] = (array_key_exists('title', $element) ? $element['title'] : 'Untitled');
$item['uri'] = self::URI . 'photo.gne?id=' . $imageID;
$description = (array_key_exists('description', $element) ? $element['description'] : '');
$item['content'] = '<a href="'
. $item['uri']
. '"><img src="'
. $imageURI
. '" /></a><br><p>'
. $description
. '</p>';
$this->items[] = $item;
break;
}
}
}
}
private function getPhotoModels($json, $filter) {
// The JSON model contains a "legend" array, where each element contains
// the path to an element in the "main" object
$photo_models = array();
foreach($json['legend'] as $legend) {
$photo_model = $json['main'];
foreach($legend as $element) { // Traverse tree
$photo_model = $photo_model[$element];
}
// We are only interested in content
if($photo_model['_flickrModelRegistry'] === $filter) {
$photo_models[] = $photo_model;
}
}
return $photo_models;
}
private function extractEnclosures($model) {
$areas = array();
foreach($model['sizes'] as $size) {
$areas[$size['width'] * $size['height']] = $size['url'];
}
return array($this->fixURL(max($areas)));
}
private function extractContentImage($model) {
$areas = array();
$limit = 320 * 240;
foreach($model['sizes'] as $size) {
$image_area = $size['width'] * $size['height'];
if($image_area >= $limit) {
$areas[$image_area] = $size['url'];
}
}
return $this->fixURL(min($areas));
}
private function fixURL($url) {
// For some reason the image URLs don't include the protocol (https)
if(strpos($url, '//') === 0) {
$url = 'https:' . $url;
}
return $url;
}
}

41
bridges/ForGifsBridge.php Executable file
View File

@@ -0,0 +1,41 @@
<?php
class ForGifsBridge extends FeedExpander {
const MAINTAINER = 'logmanoriginal';
const NAME = 'forgifs Bridge';
const URI = 'https://forgifs.com';
const DESCRIPTION = 'Returns the forgifs feed with actual gifs instead of images';
public function collectData() {
$this->collectExpandableDatas('https://forgifs.com/gallery/srss/7');
}
protected function parseItem($feedItem) {
$item = parent::parseItem($feedItem);
$content = str_get_html($item['content']);
$img = $content->find('img', 0);
$poster = $img->src;
// The actual gif is the same path but its id must be decremented by one.
// Example:
// http://forgifs.com/gallery/d/279419-2/Reporter-videobombed-shoulder-checks.gif
// http://forgifs.com/gallery/d/279418-2/Reporter-videobombed-shoulder-checks.gif
// Notice how this changes ----------^
// Now let's extract that number and do some math
// Notice: Technically we could also load the content page but that would
// require unnecessary traffic. As long as it works...
$num = substr($img->src, 29, 6);
$num -= 1;
$img->src = substr_replace($img->src, $num, 29, strlen($num));
$img->width = 'auto';
$img->height = 'auto';
$item['content'] = $content;
return $item;
}
}

View File

@@ -0,0 +1,164 @@
<?php
class GitHubGistBridge extends BridgeAbstract {
const NAME = 'GitHubGist comment bridge';
const URI = 'https://gist.github.com';
const DESCRIPTION = 'Generates feeds for Gist comments';
const MAINTAINER = 'logmanoriginal';
const CACHE_TIMEOUT = 3600;
const PARAMETERS = array(array(
'id' => array(
'name' => 'Gist',
'type' => 'text',
'required' => true,
'title' => 'Insert Gist ID or URI',
'exampleValue' => '2646763, https://gist.github.com/2646763'
)
));
private $filename;
public function getURI() {
$id = $this->getInput('id') ?: '';
$urlpath = parse_url($id, PHP_URL_PATH);
if($urlpath) {
$components = explode('/', $urlpath);
$id = end($components);
}
return static::URI . '/' . $id;
}
public function getName() {
return $this->filename ? $this->filename . ' - ' . static::NAME : static::NAME;
}
public function collectData() {
$html = getSimpleHTMLDOM($this->getURI(),
null,
null,
true,
true,
DEFAULT_TARGET_CHARSET,
false, // Do NOT remove line breaks
DEFAULT_BR_TEXT,
DEFAULT_SPAN_TEXT)
or returnServerError('Could not request ' . $this->getURI());
$html = defaultLinkTo($html, static::URI);
$fileinfo = $html->find('[class="file-info"]', 0)
or returnServerError('Could not find file info!');
$this->filename = $fileinfo->plaintext;
$comments = $html->find('div[class="timeline-comment-wrapper"]');
if(is_null($comments)) { // no comments yet
return;
}
foreach($comments as $comment) {
$uri = $comment->find('a[href^=#gistcomment]', 0)
or returnServerError('Could not find comment anchor!');
$title = $comment->find('div[class="unminimized-comment"] h3[class="timeline-comment-header-text"]', 0)
or returnServerError('Could not find comment header text!');
$datetime = $comment->find('[datetime]', 0)
or returnServerError('Could not find comment datetime!');
$author = $comment->find('a.author', 0)
or returnServerError('Could not find author name!');
$message = $comment->find('[class="comment-body"]', 0)
or returnServerError('Could not find comment body!');
$item = array();
$item['uri'] = $this->getURI() . $uri->href;
$item['title'] = str_replace('commented', 'commented on', $title->plaintext);
$item['timestamp'] = strtotime($datetime->datetime);
$item['author'] = '<a href="' . $author->href . '">' . $author->plaintext . '</a>';
$item['content'] = $this->fixContent($message);
// $item['enclosures'] = array();
// $item['categories'] = array();
$this->items[] = $item;
}
}
/** Removes all unnecessary tags and adds formatting */
private function fixContent($content){
// Restore code (inside <pre />) highlighting
foreach($content->find('pre') as $pre) {
$pre->style = <<<EOD
padding: 16px;
overflow: auto;
font-size: 85%;
line-height: 1.45;
background-color: #f6f8fa;
border-radius: 3px;
word-wrap: normal;
box-sizing: border-box;
margin-bottom: 16px;
EOD;
$code = $pre->find('code', 0);
if($code) {
$code->style = <<<EOD
white-space: pre;
word-break: normal;
EOD;
}
}
// find <code /> not inside <pre /> (`inline-code`)
foreach($content->find('code') as $code) {
if($code->parent()->tag === 'pre') {
continue;
}
$code->style = <<<EOD
background-color: rgba(27,31,35,0.05);
padding: 0.2em 0.4em;
border-radius: 3px;
EOD;
}
// restore text spacing
foreach($content->find('p') as $p) {
$p->style = 'margin-bottom: 16px;';
}
// Remove unnecessary tags
$content = strip_tags(
$content->innertext,
'<p><a><img><ol><ul><li><table><tr><th><td><string><pre><code><br><hr><h>'
);
return $content;
}
}

View File

@@ -1,12 +1,12 @@
<?php
class GooglePlusPostBridge extends BridgeAbstract{
protected $_title;
protected $_url;
private $title;
private $url;
const MAINTAINER = 'Grummfy';
const MAINTAINER = 'Grummfy, logmanoriginal';
const NAME = 'Google Plus Post Bridge';
const URI = 'https://plus.google.com/';
const URI = 'https://plus.google.com';
const CACHE_TIMEOUT = 600; //10min
const DESCRIPTION = 'Returns user public post (without API).';
@@ -14,10 +14,16 @@ class GooglePlusPostBridge extends BridgeAbstract{
'username' => array(
'name' => 'username or Id',
'required' => true
),
'include_media' => array(
'name' => 'Include media',
'type' => 'checkbox',
'title' => 'Enable to include media in the feed content'
)
));
public function collectData(){
$username = $this->getInput('username');
// Usernames start with a + if it's not an ID
@@ -25,22 +31,20 @@ class GooglePlusPostBridge extends BridgeAbstract{
$username = '+' . $username;
}
// get content parsed
$html = getSimpleHTMLDOMCached(self::URI . urlencode($username) . '/posts')
$html = getSimpleHTMLDOM(static::URI . '/' . urlencode($username) . '/posts')
or returnServerError('No results for this query.');
// get title, url, ... there is a lot of intresting stuff in meta
$this->_title = $html->find('meta[property=og:title]', 0)->getAttribute('content');
$this->_url = $html->find('meta[property=og:url]', 0)->getAttribute('content');
$html = defaultLinkTo($html, static::URI);
$this->title = $html->find('meta[property=og:title]', 0)->getAttribute('content');
$this->url = $html->find('meta[property=og:url]', 0)->getAttribute('content');
// I don't even know where to start with this discusting html...
foreach($html->find('div[jsname=WsjYwc]') as $post) {
$item = array();
$item['author'] = $item['fullname'] = $post->find('div div div div a', 0)->innertext;
$item['id'] = $post->find('div div div', 0)->getAttribute('id');
$item['avatar'] = $post->find('div img', 0)->src;
$item['uri'] = self::URI . $post->find('div div div a', 1)->href;
$item['author'] = $post->find('div div div div a', 0)->innertext;
$item['uri'] = $post->find('div div div a', 1)->href;
$timestamp = $post->find('a.qXj2He span', 0);
@@ -51,61 +55,149 @@ class GooglePlusPostBridge extends BridgeAbstract{
$timestamp->getAttribute('aria-label')));
}
// hashtag to treat : https://plus.google.com/explore/tag
// $hashtags = array();
// foreach($post->find('a.d-s') as $hashtag){
// $hashtags[trim($hashtag->plaintext)] = self::URI . $hashtag->href;
// }
$message = $post->find('div[jsname=EjRJtf]', 0);
$item['content'] = '';
// Empty messages are not supported right now
if(!$message) {
continue;
}
// avatar display
$item['content'] .= '<div style="float:left; margin: 0 0.5em 0.5em 0;"><a href="'
. self::URI
. urlencode($this->getInput('username'));
$item['content'] .= '"><img align="top" alt="'
$item['content'] = '<div style="float: left; padding: 0 10px 10px 0;"><a href="'
. $this->url
. '"><img align="top" alt="'
. $item['author']
. '" src="'
. $item['avatar']
. '" /></a></div>';
. $post->find('div img', 0)->src
. '" /></a></div><div>'
. trim(strip_tags($message, '<a><p><div><img>'))
. '</div>';
$content = $post->find('div[jsname=EjRJtf]', 0);
// extract plaintext
$item['content_simple'] = $content->plaintext;
$item['title'] = substr($item['content_simple'], 0, 72) . '...';
// XXX ugly but I don't have any idea how to do a better stuff,
// str_replace on link doesn't work as expected and ask too many checks
foreach($content->find('a') as $link) {
$hasHttp = strpos($link->href, 'http');
$hasDoubleSlash = strpos($link->href, '//');
if((!$hasHttp && !$hasDoubleSlash)
|| (false !== $hasHttp && strpos($link->href, 'http') != 0)
|| (false === $hasHttp && false !== $hasDoubleSlash && $hasDoubleSlash != 0)) {
// skipp bad link, for some hashtag or other stuff
if(strpos($link->href, '/') == 0) {
$link->href = substr($link->href, 1);
}
$link->href = self::URI . $link->href;
}
// Make title at least 50 characters long, but don't add '...' if it is shorter!
if(strlen($message->plaintext) > 50) {
$end = strpos($message->plaintext, ' ', 50);
}
$content = $content->innertext;
$item['content'] .= '<div style="margin-top: -1.5em">' . $content . '</div>';
$item['content'] = trim(strip_tags($item['content'], '<a><p><div><img>'));
if(strlen(substr($message->plaintext, 0, $end)) === strlen($message->plaintext)) {
$item['title'] = $message->plaintext;
} else {
$item['title'] = substr($message->plaintext, 0, $end) . '...';
}
$media = $post->find('[jsname="MTOxpb"]', 0);
if($media) {
$item['enclosures'] = array();
foreach($media->find('img') as $img) {
$item['enclosures'][] = $this->fixImage($img)->src;
}
if($this->getInput('include_media') === true && count($item['enclosures'] > 0)) {
$item['content'] .= '<div style="clear: both;"><a href="'
. $item['enclosures'][0]
. '"><img src="'
. $item['enclosures'][0]
. '" /></a></div>';
}
}
// Add custom parameters (only useful for JSON or Plaintext)
$item['fullname'] = $item['author'];
$item['avatar'] = $post->find('div img', 0)->src;
$item['id'] = $post->find('div div div', 0)->getAttribute('id');
$item['content_simple'] = $message->plaintext;
$this->items[] = $item;
}
}
public function getName(){
return $this->_title ?: 'Google Plus Post Bridge';
return $this->title ?: 'Google Plus Post Bridge';
}
public function getURI(){
return $this->_url ?: parent::getURI();
return $this->url ?: parent::getURI();
}
private function fixImage($img) {
// There are certain images like .gif which link to a static picture and
// get replaced dynamically via JS in the browser. If we want the "real"
// image we need to account for that.
$urlparts = parse_url($img->src);
if(array_key_exists('host', $urlparts)) {
// For some reason some URIs don't contain the scheme, assume https
if(!array_key_exists('scheme', $urlparts)) {
$urlparts['scheme'] = 'https';
}
$pathelements = explode('/', $urlparts['path']);
switch($urlparts['host']) {
case 'lh3.googleusercontent.com':
if(pathinfo(end($pathelements), PATHINFO_EXTENSION)) {
// The second to last element of the path specifies the
// image format. The URL is still valid if we remove it.
unset($pathelements[count($pathelements) - 2]);
} elseif(strrpos(end($pathelements), '=') !== false) {
// Some images go throug a proxy. For those images they
// add size information after an equal sign.
// Example: '=w530-h298-n'. Again this can safely be
// removed to get the original image.
$pathelements[count($pathelements) - 1] = substr(
end($pathelements),
0,
strrpos(end($pathelements), '=')
);
}
break;
}
$urlparts['path'] = implode('/', $pathelements);
}
$img->src = $this->build_url($urlparts);
return $img;
}
/**
* From: https://gist.github.com/Ellrion/f51ba0d40ae1d62eeae44fd1adf7b704
* slightly adjusted to work with PHP < 7.0
* @param array $parts
* @return string
*/
private function build_url(array $parts)
{
$scheme = isset($parts['scheme']) ? ($parts['scheme'] . '://') : '';
$host = isset($parts['host']) ? $parts['host'] : '';
$port = isset($parts['port']) ? (':' . $parts['port']) : '';
$user = isset($parts['user']) ? $parts['user'] : '';
$pass = isset($parts['pass']) ? (':' . $parts['pass']) : '';
$pass = ($user || $pass) ? ($pass . '@') : '';
$path = isset($parts['path']) ? $parts['path'] : '';
$query = isset($parts['query']) ? ('?' . $parts['query']) : '';
$fragment = isset($parts['fragment']) ? ('#' . $parts['fragment']) : '';
return implode('', [$scheme, $user, $pass, $host, $port, $path, $query, $fragment]);
}
}

View File

@@ -0,0 +1,370 @@
<?php
/**
* This class implements a bridge for http://www.instructables.com, supporting
* general feeds and feeds by category. Instructables doesn't support HTTPS as
* of now (23.06.2018), so all connections are insecure!
*
* Remarks:
* - For some reason it is very important to have the category URI end with a
* slash, otherwise the site defaults to the main category (i.e. Technology)!
* If you need to update the categories list, enable the 'listCategories'
* function (see comments below) and run the bridge with format=Html (see page
* source)
*/
class InstructablesBridge extends BridgeAbstract {
const NAME = 'Instructables Bridge';
const URI = 'http://www.instructables.com';
const DESCRIPTION = 'Returns general feeds and feeds by category';
const MAINTAINER = 'logmanoriginal';
const PARAMETERS = array(
'Category' => array(
'category' => array(
'name' => 'Category',
'type' => 'list',
'required' => true,
'values' => array(
'Play' => array(
'All' => '/play/',
'KNEX' => '/play/knex/',
'Offbeat' => '/play/offbeat/',
'Lego' => '/play/lego/',
'Airsoft' => '/play/airsoft/',
'Card Games' => '/play/card-games/',
'Guitars' => '/play/guitars/',
'Instruments' => '/play/instruments/',
'Magic Tricks' => '/play/magic-tricks/',
'Minecraft' => '/play/minecraft/',
'Music' => '/play/music/',
'Nerf' => '/play/nerf/',
'Nintendo' => '/play/nintendo/',
'Office Supplies' => '/play/office-supplies/',
'Paintball' => '/play/paintball/',
'Paper Airplanes' => '/play/paper-airplanes/',
'Party Tricks' => '/play/party-tricks/',
'PlayStation' => '/play/playstation/',
'Pranks and Humor' => '/play/pranks-and-humor/',
'Puzzles' => '/play/puzzles/',
'Siege Engines' => '/play/siege-engines/',
'Sports' => '/play/sports/',
'Table Top' => '/play/table-top/',
'Toys' => '/play/toys/',
'Video Games' => '/play/video-games/',
'Wii' => '/play/wii/',
'Xbox' => '/play/xbox/',
'Yo-Yo' => '/play/yo-yo/',
),
'Craft' => array(
'All' => '/craft/',
'Art' => '/craft/art/',
'Sewing' => '/craft/sewing/',
'Paper' => '/craft/paper/',
'Jewelry' => '/craft/jewelry/',
'Fashion' => '/craft/fashion/',
'Books & Journals' => '/craft/books-and-journals/',
'Cards' => '/craft/cards/',
'Clay' => '/craft/clay/',
'Duct Tape' => '/craft/duct-tape/',
'Embroidery' => '/craft/embroidery/',
'Felt' => '/craft/felt/',
'Fiber Arts' => '/craft/fiber-arts/',
'Gifts & Wrapping' => '/craft/gifts-and-wrapping/',
'Knitting & Crocheting' => '/craft/knitting-and-crocheting/',
'Leather' => '/craft/leather/',
'Mason Jars' => '/craft/mason-jars/',
'No-Sew' => '/craft/no-sew/',
'Parties & Weddings' => '/craft/parties-and-weddings/',
'Print Making' => '/craft/print-making/',
'Soap' => '/craft/soap/',
'Wallets' => '/craft/wallets/',
),
'Technology' => array(
'All' => '/technology/',
'Electronics' => '/technology/electronics/',
'Arduino' => '/technology/arduino/',
'Photography' => '/technology/photography/',
'Leds' => '/technology/leds/',
'Science' => '/technology/science/',
'Reuse' => '/technology/reuse/',
'Apple' => '/technology/apple/',
'Computers' => '/technology/computers/',
'3D Printing' => '/technology/3D-Printing/',
'Robots' => '/technology/robots/',
'Art' => '/technology/art/',
'Assistive Tech' => '/technology/assistive-technology/',
'Audio' => '/technology/audio/',
'Clocks' => '/technology/clocks/',
'CNC' => '/technology/cnc/',
'Digital Graphics' => '/technology/digital-graphics/',
'Gadgets' => '/technology/gadgets/',
'Kits' => '/technology/kits/',
'Laptops' => '/technology/laptops/',
'Lasers' => '/technology/lasers/',
'Linux' => '/technology/linux/',
'Microcontrollers' => '/technology/microcontrollers/',
'Microsoft' => '/technology/microsoft/',
'Mobile' => '/technology/mobile/',
'Raspberry Pi' => '/technology/raspberry-pi/',
'Remote Control' => '/technology/remote-control/',
'Sensors' => '/technology/sensors/',
'Software' => '/technology/software/',
'Soldering' => '/technology/soldering/',
'Speakers' => '/technology/speakers/',
'Steampunk' => '/technology/steampunk/',
'Tools' => '/technology/tools/',
'USB' => '/technology/usb/',
'Wearables' => '/technology/wearables/',
'Websites' => '/technology/websites/',
'Wireless' => '/technology/wireless/',
),
'Workshop' => array(
'All' => '/workshop/',
'Woodworking' => '/workshop/woodworking/',
'Tools' => '/workshop/tools/',
'Gardening' => '/workshop/gardening/',
'Cars' => '/workshop/cars/',
'Metalworking' => '/workshop/metalworking/',
'Cardboard' => '/workshop/cardboard/',
'Electric Vehicles' => '/workshop/electric-vehicles/',
'Energy' => '/workshop/energy/',
'Furniture' => '/workshop/furniture/',
'Home Improvement' => '/workshop/home-improvement/',
'Home Theater' => '/workshop/home-theater/',
'Hydroponics' => '/workshop/hydroponics/',
'Laser Cutting' => '/workshop/laser-cutting/',
'Lighting' => '/workshop/lighting/',
'Molds & Casting' => '/workshop/molds-and-casting/',
'Motorcycles' => '/workshop/motorcycles/',
'Organizing' => '/workshop/organizing/',
'Pallets' => '/workshop/pallets/',
'Repair' => '/workshop/repair/',
'Shelves' => '/workshop/shelves/',
'Solar' => '/workshop/solar/',
'Workbenches' => '/workshop/workbenches/',
),
'Home' => array(
'All' => '/home/',
'Halloween' => '/home/halloween/',
'Decorating' => '/home/decorating/',
'Organizing' => '/home/organizing/',
'Pets' => '/home/pets/',
'Life Hacks' => '/home/life-hacks/',
'Beauty' => '/home/beauty/',
'Christmas' => '/home/christmas/',
'Cleaning' => '/home/cleaning/',
'Education' => '/home/education/',
'Finances' => '/home/finances/',
'Gardening' => '/home/gardening/',
'Green' => '/home/green/',
'Health' => '/home/health/',
'Hiding Places' => '/home/hiding-places/',
'Holidays' => '/home/holidays/',
'Homesteading' => '/home/homesteading/',
'Kids' => '/home/kids/',
'Kitchen' => '/home/kitchen/',
'Life Skills' => '/home/life-skills/',
'Parenting' => '/home/parenting/',
'Pest Control' => '/home/pest-control/',
'Relationships' => '/home/relationships/',
'Reuse' => '/home/reuse/',
'Travel' => '/home/travel/',
),
'Outside' => array(
'All' => '/outside/',
'Bikes' => '/outside/bikes/',
'Survival' => '/outside/survival/',
'Backyard' => '/outside/backyard/',
'Beach' => '/outside/beach/',
'Birding' => '/outside/birding/',
'Boats' => '/outside/boats/',
'Camping' => '/outside/camping/',
'Climbing' => '/outside/climbing/',
'Fire' => '/outside/fire/',
'Fishing' => '/outside/fishing/',
'Hunting' => '/outside/hunting/',
'Kites' => '/outside/kites/',
'Knives' => '/outside/knives/',
'Knots' => '/outside/knots/',
'Paracord' => '/outside/paracord/',
'Rockets' => '/outside/rockets/',
'Skateboarding' => '/outside/skateboarding/',
'Snow' => '/outside/snow/',
'Water' => '/outside/water/',
),
'Food' => array(
'All' => '/food/',
'Dessert' => '/food/dessert/',
'Snacks & Appetizers' => '/food/snacks-and-appetizers/',
'Bacon' => '/food/bacon/',
'BBQ & Grilling' => '/food/bbq-and-grilling/',
'Beverages' => '/food/beverages/',
'Bread' => '/food/bread/',
'Breakfast' => '/food/breakfast/',
'Cake' => '/food/cake/',
'Candy' => '/food/candy/',
'Canning & Preserves' => '/food/canning-and-preserves/',
'Cocktails & Mocktails' => '/food/cocktails-and-mocktails/',
'Coffee' => '/food/coffee/',
'Cookies' => '/food/cookies/',
'Cupcakes' => '/food/cupcakes/',
'Homebrew' => '/food/homebrew/',
'Main Course' => '/food/main-course/',
'Pasta' => '/food/pasta/',
'Pie' => '/food/pie/',
'Pizza' => '/food/pizza/',
'Salad' => '/food/salad/',
'Sandwiches' => '/food/sandwiches/',
'Soups & Stews' => '/food/soups-and-stews/',
'Vegetarian & Vegan' => '/food/vegetarian-and-vegan/',
),
'Costumes' => array(
'All' => '/costumes/',
'Props' => '/costumes/props-and-accessories/',
'Animals' => '/costumes/animals/',
'Comics' => '/costumes/comics/',
'Fantasy' => '/costumes/fantasy/',
'For Kids' => '/costumes/for-kids/',
'For Pets' => '/costumes/for-pets/',
'Funny' => '/costumes/funny/',
'Games' => '/costumes/games/',
'Historic & Futuristic' => '/costumes/historic-and-futuristic/',
'Makeup' => '/costumes/makeup/',
'Masks' => '/costumes/masks/',
'Scary' => '/costumes/scary/',
'TV & Movies' => '/costumes/tv-and-movies/',
'Weapons & Armor' => '/costumes/weapons-and-armor/',
)
),
'title' => 'Select your category (required)',
'defaultValue' => 'Technology'
),
'filter' => array(
'name' => 'Filter',
'type' => 'list',
'required' => true,
'values' => array(
'Featured' => ' ',
'Recent' => 'recent/',
'Popular' => 'popular/',
'Views' => 'views/',
'Contest Winners' => 'winners/'
),
'title' => 'Select a filter',
'defaultValue' => 'Featured'
)
)
);
private $uri;
public function collectData() {
// Enable the following line to get the category list (dev mode)
// $this->listCategories();
$this->uri = static::URI;
switch($this->queriedContext) {
case 'Category': $this->uri .= $this->getInput('category') . $this->getInput('filter');
}
$html = getSimpleHTMLDOM($this->uri)
or returnServerError('Error loading category ' . $this->uri);
foreach($html->find('ul.explore-covers-list li') as $cover) {
$item = array();
$item['uri'] = static::URI . $cover->find('a.cover-image', 0)->href;
$item['title'] = $cover->find('.title', 0)->innertext;
$item['author'] = $this->getCategoryAuthor($cover);
$item['content'] = '<a href='
. $item['uri']
. '><img src='
. $cover->find('a.cover-image img', 0)->src
. '></a>';
$image = str_replace('.RECTANGLE1', '.LARGE', $cover->find('a.cover-image img', 0)->src);
$item['enclosures'] = [$image];
$this->items[] = $item;
}
}
public function getName() {
if(!is_null($this->getInput('category'))
&& !is_null($this->getInput('filter'))) {
foreach(self::PARAMETERS[$this->queriedContext]['category']['values'] as $key => $value) {
$subcategory = array_search($this->getInput('category'), $value);
if($subcategory !== false)
break;
}
$filter = array_search(
$this->getInput('filter'),
self::PARAMETERS[$this->queriedContext]['filter']['values']
);
return $subcategory . ' (' . $filter . ') - ' . static::NAME;
}
return parent::getName();
}
public function getURI() {
if(!is_null($this->getInput('category'))
&& !is_null($this->getInput('filter'))) {
return $this->uri;
}
return parent::getURI();
}
/**
* Returns a list of categories for development purposes (used to build the
* parameters list)
*/
private function listCategories(){
// Use arbitrary category to receive full list
$html = getSimpleHTMLDOM(self::URI . '/technology/');
foreach($html->find('.channel a') as $channel) {
$name = html_entity_decode(trim($channel->innertext));
// Remove unwanted entities
$name = str_replace("'", '', $name);
$name = str_replace('&#39;', '', $name);
$uri = $channel->href;
$category = explode('/', $uri)[1];
if(!isset($categories)
|| !array_key_exists($category, $categories)
|| !in_array($uri, $categories[$category]))
$categories[$category][$name] = $uri;
}
// Build PHP array manually
foreach($categories as $key => $value) {
$name = ucfirst($key);
echo "'{$name}' => array(\n";
echo "\t'All' => '/{$key}/',\n";
foreach($value as $name => $uri) {
echo "\t'{$name}' => '{$uri}',\n";
}
echo "),\n";
}
die;
}
/**
* Returns the author as anchor for a given cover.
*/
private function getCategoryAuthor($cover) {
return '<a href='
. static::URI . $cover->find('span.author a', 0)->href
. '>'
. $cover->find('span.author a', 0)->innertext
. '</a>';
}
}

View File

@@ -42,6 +42,7 @@ class LeBonCoinBridge extends BridgeAbstract {
'Réunion' => '26'
)
),
'cities' => array('name' => 'Ville'),
'c' => array(
'name' => 'Catégorie',
'type' => 'list',
@@ -154,6 +155,7 @@ class LeBonCoinBridge extends BridgeAbstract {
$params = array(
'text' => $this->getInput('k'),
'region' => $this->getInput('r'),
'cities' => $this->getInput('cities'),
'category' => $this->getInput('c'),
'owner_type' => $this->getInput('o'),
);

825
bridges/SkimfeedBridge.php Normal file
View File

@@ -0,0 +1,825 @@
<?php
class SkimfeedBridge extends BridgeAbstract {
const CONTEXT_NEWS_BOX = 'News box';
const CONTEXT_HOT_TOPICS = 'Hot topics';
const CONTEXT_TECH_NEWS = 'Tech news';
const CONTEXT_CUSTOM = 'Custom feed';
const NAME = 'Skimfeed Bridge';
const URI = 'https://skimfeed.com';
const DESCRIPTION = 'Returns feeds from Skimfeed, also supports custom feeds!';
const MAINTAINER = 'logmanoriginal';
const CACHE_TIMEOUT = 3600;
const PARAMETERS = array(
self::CONTEXT_NEWS_BOX => array( // auto-generated (see below)
'box_channel' => array(
'name' => 'Channel',
'type' => 'list',
'required' => true,
'title' => 'Select your channel',
'values' => array(
'Hacker News' => '/news/hacker-news.html',
'QZ' => '/news/qz.html',
'The Verge' => '/news/the-verge.html',
'Slashdot' => '/news/slashdot.html',
'Lifehacker' => '/news/lifehacker.html',
'Gizmag' => '/news/gizmag.html',
'Fast Company' => '/news/fast-company.html',
'Engadget' => '/news/engadget.html',
'Wired' => '/news/wired.html',
'MakeUseOf' => '/news/makeuseof.html',
'Techcrunch' => '/news/techcrunch.html',
'Apple Insider' => '/news/apple-insider.html',
'ArsTechnica' => '/news/arstechnica.html',
'Tech in Asia' => '/news/tech-in-asia.html',
'FastCoExist' => '/news/fastcoexist.html',
'Digital Trends' => '/news/digital-trends.html',
'AnandTech' => '/news/anandtech.html',
'How to Geek' => '/news/how-to-geek.html',
'Geek' => '/news/geek.html',
'BBC Technology' => '/news/bbc-technology.html',
'Extreme Tech' => '/news/extreme-tech.html',
'Packet Storm Sec' => '/news/packet-storm-sec.html',
'MedGadget' => '/news/medgadget.html',
'Design' => '/news/design.html',
'The Next Web' => '/news/the-next-web.html',
'Bit-Tech' => '/news/bit-tech.html',
'Next Big Future' => '/news/next-big-future.html',
'A VC' => '/news/a-vc.html',
'Copyblogger' => '/news/copyblogger.html',
'Smashing Mag' => '/news/smashing-mag.html',
'Continuations' => '/news/continuations.html',
'Cult of Mac' => '/news/cult-of-mac.html',
'SecuriTeam' => '/news/securiteam.html',
'The Tech Block' => '/news/the-tech-block.html',
'BetaBeat' => '/news/betabeat.html',
'PC Mag' => '/news/pc-mag.html',
'Venture Beat' => '/news/venture-beat.html',
'ReadWriteWeb' => '/news/readwriteweb.html',
'High Scalability' => '/news/high-scalability.html',
)
)
),
self::CONTEXT_HOT_TOPICS => array(),
self::CONTEXT_TECH_NEWS => array( // auto-generated (see below)
'tech_channel' => array(
'name' => 'Tech channel',
'type' => 'list',
'required' => true,
'title' => 'Select your tech channel',
'values' => array(
'Agg' => array(
'Reddit' => '/news/reddit.html',
'Tech Insider' => '/news/tech-insider.html',
'Digg' => '/news/digg.html',
'Meta Filter' => '/news/meta-filter.html',
'Fark' => '/news/fark.html',
'Mashable' => '/news/mashable.html',
'Ad Week' => '/news/ad-week.html',
'The Chive' => '/news/the-chive.html',
'BoingBoing' => '/news/boingboing.html',
'Vice' => '/news/vice.html',
'ClientsFromHell' => '/news/clientsfromhell.html',
'How Stuff Works' => '/news/how-stuff-works.html',
'Buzzfeed' => '/news/buzzfeed.html',
'BoingBoing' => '/news/boingboing.html',
'Cracked' => '/news/cracked.html',
'Weird News' => '/news/weird-news.html',
'ITOTD' => '/news/itotd.html',
'Metafilter' => '/news/metafilter.html',
'TheOnion' => '/news/theonion.html',
),
'Cars' => array(
'Reddit Cars' => '/news/reddit-cars.html',
'NYT Auto' => '/news/nyt-auto.html',
'Truth About Cars' => '/news/truth-about-cars.html',
'AutoBlog' => '/news/autoblog.html',
'AutoSpies' => '/news/autospies.html',
'Autoweek' => '/news/autoweek.html',
'The Garage' => '/news/the-garage.html',
'Car and Driver' => '/news/car-and-driver.html',
'EGM Car Tech' => '/news/egm-car-tech.html',
'Top Gear' => '/news/top-gear.html',
'eGarage' => '/news/egarage.html',
),
'Comics' => array(
'Penny Arcade' => '/news/penny-arcade.html',
'XKCD' => '/news/xkcd.html',
'Channelate' => '/news/channelate.html',
'Savage Chicken' => '/news/savage-chicken.html',
'Dinosaur Comics' => '/news/dinosaur-comics.html',
'Explosm' => '/news/explosm.html',
'PoorlyDLines' => '/news/poorlydlines.html',
'Moonbeard' => '/news/moonbeard.html',
'Nedroid' => '/news/nedroid.html',
),
'Design' => array(
'FastCoCreate' => '/news/fastcocreate.html',
'Dezeen' => '/news/dezeen.html',
'Design Boom' => '/news/design-boom.html',
'Mmminimal' => '/news/mmminimal.html',
'We Heart' => '/news/we-heart.html',
'CreativeBloq' => '/news/creativebloq.html',
'TheDSGNblog' => '/news/thedsgnblog.html',
'Grainedit' => '/news/grainedit.html',
),
'Football' => array(
'Mail Football' => '/news/mail-football.html',
'Yahoo Football' => '/news/yahoo-football.html',
'FourFourTwo' => '/news/fourfourtwo.html',
'Goal' => '/news/goal.html',
'BBC Football' => '/news/bbc-football.html',
'TalkSport' => '/news/talksport.html',
'101 Great Goals' => '/news/101-great-goals.html',
'Who Scored' => '/news/who-scored.html',
'Football365 Champ' => '/news/football365-champ.html',
'Football365 Premier' => '/news/football365-premier.html',
'BleacherReport' => '/news/bleacherreport.html',
),
'Gaming' => array(
'Polygon' => '/news/polygon.html',
'Gamespot' => '/news/gamespot.html',
'RockPaperShotgun' => '/news/rockpapershotgun.html',
'VG247' => '/news/vg247.html',
'IGN' => '/news/ign.html',
'Reddit Games' => '/news/reddit-games.html',
'TouchArcade' => '/news/toucharcade.html',
'GamesRadar' => '/news/gamesradar.html',
'Siliconera' => '/news/siliconera.html',
'Reddit GameDeals' => '/news/reddit-gamedeals.html',
'Joystiq' => '/news/joystiq.html',
'GameInformer' => '/news/gameinformer.html',
'PSN Blog' => '/news/psn-blog.html',
'Reddit GamerNews' => '/news/reddit-gamernews.html',
'Steam' => '/news/steam.html',
'DualShockers' => '/news/dualshockers.html',
'ShackNews' => '/news/shacknews.html',
'CheapAssGamer' => '/news/cheapassgamer.html',
'Eurogamer' => '/news/eurogamer.html',
'Major Nelson' => '/news/major-nelson.html',
'Reddit Truegaming' => '/news/reddit-truegaming.html',
'GameTrailers' => '/news/gametrailers.html',
'GamaSutra' => '/news/gamasutra.html',
'USGamer' => '/news/usgamer.html',
'Shoryuken' => '/news/shoryuken.html',
'Destructoid' => '/news/destructoid.html',
'ArsGaming' => '/news/arsgaming.html',
'XBOX Blog' => '/news/xbox-blog.html',
'GiantBomb' => '/news/giantbomb.html',
'VideoGamer' => '/news/videogamer.html',
'Pocket Tactics' => '/news/pocket-tactics.html',
'WiredGaming' => '/news/wiredgaming.html',
'AllGamesBeta' => '/news/allgamesbeta.html',
'OnGamers' => '/news/ongamers.html',
'Reddit GameBundles' => '/news/reddit-gamebundles.html',
'Kotaku' => '/news/kotaku.html',
'PCGamer' => '/news/pcgamer.html',
),
'Investing' => array(
'Seeking Alpha' => '/news/seeking-alpha.html',
'BBC Business' => '/news/bbc-business.html',
'Harvard Biz' => '/news/harvard-biz.html',
'Market Watch' => '/news/market-watch.html',
'Investor Place' => '/news/investor-place.html',
'Money Week' => '/news/money-week.html',
'Moneybeat' => '/news/moneybeat.html',
'Dealbook' => '/news/dealbook.html',
'Economist Business' => '/news/economist-business.html',
'Economist' => '/news/economist.html',
'Economist CN' => '/news/economist-cn.html',
),
'Long' => array(
'The Atlantic' => '/news/the-atlantic.html',
'Reddit Long' => '/news/reddit-long.html',
'Paris Review' => '/news/paris-review.html',
'New Yorker' => '/news/new-yorker.html',
'LongForm' => '/news/longform.html',
'LongReads' => '/news/longreads.html',
'The Browser' => '/news/the-browser.html',
'The Feature' => '/news/the-feature.html',
),
'MMA' => array(
'MMA Weekly' => '/news/mma-weekly.html',
'MMAFighting' => '/news/mmafighting.html',
'Reddit MMA' => '/news/reddit-mma.html',
'Sherdog Articles' => '/news/sherdog-articles.html',
'FightLand Vice' => '/news/fightland-vice.html',
'Sherdog Forum' => '/news/sherdog-forum.html',
'MMA Junkie' => '/news/mma-junkie.html',
'Sherdog MMA Video' => '/news/sherdog-mma-video.html',
'BloodyElbow' => '/news/bloodyelbow.html',
'CageWriter' => '/news/cagewriter.html',
'Sherdog News' => '/news/sherdog-news.html',
'MMAForum' => '/news/mmaforum.html',
'MMA Junkie Radio' => '/news/mma-junkie-radio.html',
'UFC News' => '/news/ufc-news.html',
'FightLinker' => '/news/fightlinker.html',
'Bodybuilding MMA' => '/news/bodybuilding-mma.html',
'BleacherReport MMA' => '/news/bleacherreport-mma.html',
'FiveOuncesofPain' => '/news/fiveouncesofpain.html',
'Sherdog Pictures' => '/news/sherdog-pictures.html',
'CagePotato' => '/news/cagepotato.html',
'Sherdog Radio' => '/news/sherdog-radio.html',
'ProMMARadio' => '/news/prommaradio.html',
),
'Mobile' => array(
'Macrumors' => '/news/macrumors.html',
'Android Police' => '/news/android-police.html',
'GSM Arena' => '/news/gsm-arena.html',
'DigiTrend Mobile' => '/news/digitrend-mobile.html',
'Mobile Nation' => '/news/mobile-nation.html',
'TechRadar' => '/news/techradar.html',
'ZDNET Mobile' => '/news/zdnet-mobile.html',
'MacWorld' => '/news/macworld.html',
'Android Dev Blog' => '/news/android-dev-blog.html',
),
'News' => array(
'Daily Mail' => '/news/daily-mail.html',
'Business Insider' => '/news/business-insider.html',
'The Guardian' => '/news/the-guardian.html',
'Fox' => '/news/fox.html',
'BBC World' => '/news/bbc-world.html',
'MSNBC' => '/news/msnbc.html',
'ABC News' => '/news/abc-news.html',
'Al Jazeera' => '/news/al-jazeera.html',
'Business Insider India' => '/news/business-insider-india.html',
'Observer' => '/news/observer.html',
'NYT Tech' => '/news/nyt-tech.html',
'NYT World' => '/news/nyt-world.html',
'CNN' => '/news/cnn.html',
'Japan Times' => '/news/japan-times.html',
'WorldCrunch' => '/news/worldcrunch.html',
'Pro publica' => '/news/pro-publica.html',
'OZY' => '/news/ozy.html',
'Times of India' => '/news/times-of-india.html',
'The Australian' => '/news/the-australian.html',
'Harpers' => '/news/harpers.html',
'Moscow Times' => '/news/moscow-times.html',
'The Times' => '/news/the-times.html',
'Reuters Tech' => '/news/reuters-tech.html',
),
'Politics' => array(
'FreeRepublic' => '/news/freerepublic.html',
'Salon' => '/news/salon.html',
'DrudgeReport' => '/news/drudgereport.html',
'TheHill' => '/news/thehill.html',
'TheBlaze' => '/news/theblaze.html',
'InfoWars' => '/news/infowars.html',
'New Republic' => '/news/new-republic.html',
'WashTimes' => '/news/washtimes.html',
'RealCleanPol' => '/news/realcleanpol.html',
'Fact Check' => '/news/fact-check.html',
'DailyKos' => '/news/dailykos.html',
'NewsMax' => '/news/newsmax.html',
'Politico' => '/news/politico.html',
'Michelle Malkin' => '/news/michelle-malkin.html',
),
'Reddit' => array(
'R Movies' => '/news/r-movies.html',
'R News' => '/news/r-news.html',
'Futurology' => '/news/futurology.html',
'R All' => '/news/r-all.html',
'R Music' => '/news/r-music.html',
'R Askscience' => '/news/r-askscience.html',
'R Technology' => '/news/r-technology.html',
'R Bestof' => '/news/r-bestof.html',
'R Askreddit' => '/news/r-askreddit.html',
'R Worldnews' => '/news/r-worldnews.html',
'R Explainlikeimfive' => '/news/r-explainlikeimfive.html',
'R Iama' => '/news/r-iama.html',
),
'Science' => array(
'PhysOrg' => '/news/physorg.html',
'Hack-a-day' => '/news/hack-a-day.html',
'Reddit Science' => '/news/reddit-science.html',
'Stats Blog' => '/news/stats-blog.html',
'Flowing Data' => '/news/flowing-data.html',
'Eureka Alert' => '/news/eureka-alert.html',
'Robotics BizRev' => '/news/robotics-bizrev.html',
'Planet big Data' => '/news/planet-big-data.html',
'Makezine' => '/news/makezine.html',
'MIT Tech' => '/news/mit-tech.html',
'R Bloggers' => '/news/r-bloggers.html',
'DataIsBeautiful' => '/news/dataisbeautiful.html',
'Ted Videos' => '/news/ted-videos.html',
'Advanced Science' => '/news/advanced-science.html',
'Robotiq' => '/news/robotiq.html',
'Science Daily' => '/news/science-daily.html',
'IEEE Robotics' => '/news/ieee-robotics.html',
'PSFK' => '/news/psfk.html',
'Discover Magazine' => '/news/discover-magazine.html',
'DataTau' => '/news/datatau.html',
'RoboHub' => '/news/robohub.html',
'Discovery' => '/news/discovery.html',
'Smart Data' => '/news/smart-data.html',
'Whats Big Data' => '/news/whats-big-data.html',
),
'Tech' => array(
'Hacker News' => '/news/hacker-news.html',
'The Verge' => '/news/the-verge.html',
'Lifehacker' => '/news/lifehacker.html',
'Fast Company' => '/news/fast-company.html',
'ArsTechnica' => '/news/arstechnica.html',
'MakeUseOf' => '/news/makeuseof.html',
'FastCoExist' => '/news/fastcoexist.html',
'How to Geek' => '/news/how-to-geek.html',
'The Next Web' => '/news/the-next-web.html',
'Engadget' => '/news/engadget.html',
'Gizmag' => '/news/gizmag.html',
'QZ' => '/news/qz.html',
'Wired' => '/news/wired.html',
'Techcrunch' => '/news/techcrunch.html',
'Slashdot' => '/news/slashdot.html',
'Extreme Tech' => '/news/extreme-tech.html',
'AnandTech' => '/news/anandtech.html',
'Digital Trends' => '/news/digital-trends.html',
'Next Big Future' => '/news/next-big-future.html',
'Apple Insider' => '/news/apple-insider.html',
'Geek' => '/news/geek.html',
'BBC Technology' => '/news/bbc-technology.html',
'Bit-Tech' => '/news/bit-tech.html',
'Packet Storm Sec' => '/news/packet-storm-sec.html',
'Design' => '/news/design.html',
'High Scalability' => '/news/high-scalability.html',
'Smashing Mag' => '/news/smashing-mag.html',
'The Tech Block' => '/news/the-tech-block.html',
'A VC' => '/news/a-vc.html',
'Tech in Asia' => '/news/tech-in-asia.html',
'ReadWriteWeb' => '/news/readwriteweb.html',
'PC Mag' => '/news/pc-mag.html',
'Continuations' => '/news/continuations.html',
'Copyblogger' => '/news/copyblogger.html',
'Cult of Mac' => '/news/cult-of-mac.html',
'BetaBeat' => '/news/betabeat.html',
'MedGadget' => '/news/medgadget.html',
'SecuriTeam' => '/news/securiteam.html',
'Venture Beat' => '/news/venture-beat.html',
),
'Trend' => array(
'Trend Hunter' => '/news/trend-hunter.html',
'ApartmentT' => '/news/apartmentt.html',
'GQ' => '/news/gq.html',
'Digital Trends' => '/news/digital-trends.html',
'Cool Hunting' => '/news/cool-hunting.html',
'FastCoDesign' => '/news/fastcodesign.html',
'TC Startups' => '/news/tc-startups.html',
'Killer Startups' => '/news/killer-startups.html',
'DigiInfo' => '/news/digiinfo.html',
'New Startups' => '/news/new-startups.html',
'DigiTrends' => '/news/digitrends.html',
),
'Watches' => array(
'Hodinkee' => '/news/hodinkee.html',
'Quill and Pad' => '/news/quill-and-pad.html',
'Monochrome' => '/news/monochrome.html',
'Deployant' => '/news/deployant.html',
'Watches by SJX' => '/news/watches-by-sjx.html',
'Fratello Watches' => '/news/fratello-watches.html',
'A Blog to Watch' => '/news/a-blog-to-watch.html',
'Wound for Life' => '/news/wound-for-life.html',
'Watch Paper' => '/news/watch-paper.html',
'Watch Report' => '/news/watch-report.html',
'Perpetuelle' => '/news/perpetuelle.html',
),
'Youtube' => array(
'LinusTechTips' => '/news/linustechtips.html',
'MetalJesusRocks' => '/news/metaljesusrocks.html',
'TotalBiscuit' => '/news/totalbiscuit.html',
'DexBonus' => '/news/dexbonus.html',
'Lon Siedman' => '/news/lon-siedman.html',
'MKBHD' => '/news/mkbhd.html',
'Terry A Davis' => '/news/terry-a-davis.html',
'HappyConsole' => '/news/happyconsole.html',
'Austin Evans' => '/news/austin-evans.html',
'NCIX' => '/news/ncix.html',
),
)
),
),
self::CONTEXT_CUSTOM => array(
'config' => array(
'name' => 'Configuration',
'type' => 'text',
'required' => true,
'title' => 'Enter feed numbers from Skimfeed!',
'exampleValue' => '5,8,2,l,p,9,23'
)
),
'global' => array(
'limit' => array(
'name' => 'Limit',
'type' => 'number',
'title' => 'Limits the number of returned items in the feed',
'exampleValue' => 10
)
)
);
public function getURI() {
switch($this->queriedContext) {
case self::CONTEXT_NEWS_BOX:
$channel = $this->getInput('box_channel');
if($channel) {
return static::URI . $channel;
}
break;
case self::CONTEXT_HOT_TOPICS:
return static::URI;
case self::CONTEXT_TECH_NEWS:
$channel = $this->getInput('tech_channel');
if($channel) {
return static::URI . $channel;
}
break;
case self::CONTEXT_CUSTOM:
$config = $this->getInput('config');
return static::URI . '/custom.php?f=' . urlencode($config);
}
return parent::getURI();
}
public function getName() {
switch($this->queriedContext) {
case self::CONTEXT_NEWS_BOX:
$channel = $this->getInput('box_channel');
$title = array_search(
$channel,
static::PARAMETERS[self::CONTEXT_NEWS_BOX]['box_channel']['values']
);
return $title . ' - ' . static::NAME;
case self::CONTEXT_HOT_TOPICS:
return 'Hot topics - ' . static::NAME;
case self::CONTEXT_TECH_NEWS:
$channel = $this->getInput('tech_channel');
$titles = array();
foreach(static::PARAMETERS[self::CONTEXT_TECH_NEWS]['tech_channel']['values'] as $ch) {
$titles = array_merge($titles, $ch);
}
$title = array_search($channel, $titles);
return $title . ' - ' . static::NAME;
case self::CONTEXT_CUSTOM:
return 'Custom - ' . static::NAME;
}
return parent::getName();
}
public function collectData() {
// enable to export parameter lists
// $this->exportBoxChannels(); die;
// $this->exportTechChannels(); die;
$html = getSimpleHTMLDOM($this->getURI())
or returnServerError('Request to ' . $this->getURI() . ' failed!');
defaultLinkTo($html, static::URI);
switch($this->queriedContext) {
case self::CONTEXT_NEWS_BOX:
$author = array_search(
$this->getInput('box_channel'),
static::PARAMETERS[self::CONTEXT_NEWS_BOX]['box_channel']['values']
);
$author = '<a href="'
. $this->getURI()
. '">'
. $author
. '</a>';
$this->extractFeed($html, $author);
break;
case self::CONTEXT_HOT_TOPICS:
$this->extractHotTopics($html);
break;
case self::CONTEXT_TECH_NEWS:
$authors = array();
foreach(static::PARAMETERS[self::CONTEXT_TECH_NEWS]['tech_channel']['values'] as $ch) {
$authors = array_merge($authors, $ch);
}
$author = '<a href="'
. $this->getURI()
. '">'
. array_search($this->getInput('tech_channel'), $authors)
. '</a>';
$this->extractFeed($html, $author);
break;
case self::CONTEXT_CUSTOM:
$this->extractCustomFeed($html);
break;
}
}
private function extractFeed($html, $author) {
$articles = $html->find('li')
or returnServerError('Could not find articles!');
if(count($articles) === 1
&& stristr($articles[0]->plaintext, 'Nothing new in the last 48 hours')) {
return; // Nothing to show
}
$limit = $this->getInput('limit') ?: -1;
foreach($articles as $article) {
$anchor = $article->find('a', 0)
or returnServerError('Could not find anchor!');
$item = array();
$item['uri'] = $this->getTarget($anchor);
$item['title'] = trim($anchor->plaintext);
// The timestamp is encoded as relative time (max. the last 48 hours)
// like this: "- 7 hours". It should always be at the end of the article:
$age = substr($article->plaintext, strrpos($article->plaintext, '-'));
$item['timestamp'] = strtotime($age);
$item['author'] = $author;
$this->items[] = $item;
if($limit > 0 && count($this->items) >= $limit) {
return;
}
}
}
private function extractHotTopics($html) {
$topics = $html->find('#popbox ul li')
or returnServerError('Could not find topics!');
$limit = $this->getInput('limit') ?: -1;
foreach($topics as $topic) {
$anchor = $topic->find('a', 0)
or returnServerError('Could not find anchor!');
$item = array();
$item['uri'] = $this->getTarget($anchor);
$item['title'] = $anchor->title;
$this->items[] = $item;
if($limit > 0 && count($this->items) >= $limit) {
return;
}
}
}
private function extractCustomFeed($html) {
$boxes = $html->find('#boxx .boxes')
or returnServerError('Could not find boxes!');
foreach($boxes as $box) {
$anchor = $box->find('span.boxtitles a', 0)
or returnServerError('Could not find box anchor!');
$author = '<a href="' . $anchor->href . '">' . trim($anchor->plaintext) . '</a>';
$uri = $anchor->href;
$box_html = getSimpleHTMLDOM($uri)
or returnServerError('Could not load custom feed!');
$this->extractFeed($box_html, $author);
}
}
private function getTarget($anchor) {
// Anchors are linked to Skimfeed, luckily the target URI is encoded
// in that URI via '&u=<URI>':
$query = parse_url($anchor->href, PHP_URL_QUERY);
foreach(explode('&', $query) as $parameter) {
list($key, $value) = explode('=', $parameter);
if($key !== 'u') {
continue;
}
return urldecode($value);
}
}
/**
* dev-mode!
* Requires '&format=Html'
*
* Returns the 'box' array from the source site
*/
private function exportBoxChannels() {
$html = getSimpleHTMLDOMCached(static::URI)
or returnServerError('No contents received from Skimfeed!');
if(!$this->isCompatible($html)) {
returnServerError('Skimfeed version is not compatible!');
}
$boxes = $html->find('#boxx .boxes')
or returnServerError('Could not find boxes!');
// begin of 'channel' list
$message = <<<EOD
'box_channel' => array(
'name' => 'Channel',
'type' => 'list',
'required' => true,
'title' => 'Select your channel',
'values' => array(
EOD;
foreach($boxes as $box) {
$anchor = $box->find('span.boxtitles a', 0)
or returnServerError('Could not find box anchor!');
$title = trim($anchor->plaintext);
$uri = $anchor->href;
// add value
$message .= "\t\t'{$title}' => '{$uri}', \n";
}
// end of 'box' list
$message .= <<<EOD
)
),
EOD;
echo <<<EOD
<!DOCTYPE html>
<html>
<body>
<code style="white-space: pre-wrap;">{$message}</code>
</body>
</html>
EOD;
}
/**
* dev-mode!
* Requires '&format=Html'
*
* Returns the 'techs' array from the source site
*/
private function exportTechChannels() {
$html = getSimpleHTMLDOMCached(static::URI)
or returnServerError('No contents received from Skimfeed!');
if(!$this->isCompatible($html)) {
returnServerError('Skimfeed version is not compatible!');
}
$channels = $html->find('#menubar a')
or returnServerError('Could not find channels!');
// begin of 'tech_channel' list
$message = <<<EOD
'tech_channel' => array(
'name' => 'Tech channel',
'type' => 'list',
'required' => true,
'title' => 'Select your tech channel',
'values' => array(
EOD;
foreach($channels as $channel) {
if($channel->href === '#'
|| $channel->class === 'homelink'
|| $channel->plaintext === 'Twitter'
|| $channel->plaintext === 'Weather'
|| $channel->plaintext === '+Custom') {
continue;
}
$title = trim($channel->plaintext);
$uri = '/' . $channel->href;
$message .= "\t\t'{$title}' => array(\n";
$channel_html = getSimpleHTMLDOMCached(static::URI . $uri)
or returnServerError('Could not load tech channel ' . $channel->plaintext . '!');
$boxes = $channel_html->find('#boxx .boxes')
or returnServerError('Could not find boxes!');
foreach($boxes as $box) {
$anchor = $box->find('span.boxtitles a', 0)
or returnServerError('Could not find box anchor!');
$boxtitle = trim($anchor->plaintext);
$boxuri = $anchor->href;
$message .= "\t\t\t'{$boxtitle}' => '{$boxuri}', \n";
}
$message .= "\t\t),\n";
}
// end of 'box' list
$message .= <<<EOD
)
),
EOD;
echo <<<EOD
<!DOCTYPE html>
<html>
<body>
<code style="white-space: pre-wrap;">{$message}</code>
</body>
</html>
EOD;
}
/**
* Checks if the reported skimfeed version is compatible
*/
private function isCompatible($html) {
$title = $html->find('title', 0);
if(!$title) {
return false;
}
if($title->plaintext === 'Skimfeed V5.5 - Tech News') {
return true;
}
return false;
}
}

View File

@@ -1,102 +0,0 @@
<?php
class Torrent9Bridge extends BridgeAbstract {
const MAINTAINER = 'lagaisse';
const NAME = 'Torrent9 Bridge';
const URI = 'http://www.torrent9.pe';
const CACHE_TIMEOUT = 86400; // 24h = 86400s
const DESCRIPTION = 'Returns latest torrents';
const PAGE_SERIES = 'torrents_series';
const PAGE_SERIES_VOSTFR = 'torrents_series_vostfr';
const PAGE_SERIES_FR = 'torrents_series_french';
const PARAMETERS = array(
'From search' => array(
'q' => array(
'name' => 'Search',
'required' => true,
'title' => 'Type your search'
)
),
'By page' => array(
'page' => array(
'name' => 'Page',
'type' => 'list',
'required' => false,
'values' => array(
'Series' => self::PAGE_SERIES,
'Series VOST' => self::PAGE_SERIES_VOSTFR,
'Series FR' => self::PAGE_SERIES_FR,
),
'defaultValue' => self::PAGE_SERIES
)
)
);
public function collectData(){
if($this->queriedContext === 'From search') {
$request = str_replace(' ', '-', trim($this->getInput('q')));
$page = self::URI . '/search_torrent/' . urlencode($request) . '.html';
} else {
$request = $this->getInput('page');
$page = self::URI . '/' . $request . '.html';
}
$html = getSimpleHTMLDOM($page)
or returnServerError('No results for this query.');
foreach($html->find('table', 0)->find('tr') as $episode) {
if($episode->parent->tag == 'tbody') {
$urlepisode = self::URI . $episode->find('a', 0)->getAttribute('href');
//30 years = forever
$htmlepisode = getSimpleHTMLDOMCached($urlepisode, 86400 * 366 * 30);
$item = array();
$item['author'] = $episode->find('a', 0)->text();
$item['title'] = $episode->find('a', 0)->text();
$item['id'] = $episode->find('a', 0)->getAttribute('href');
$item['pubdate'] = $this->getCachedDate($urlepisode);
$textefiche = $htmlepisode->find('.movie-information', 0)->find('p', 1);
if(isset($textefiche)) {
$item['content'] = $textefiche->text();
} else {
$p = $htmlepisode->find('.movie-information', 0)->find('p');
if(!empty($p)) {
$item['content'] = $htmlepisode->find('.movie-information', 0)->find('p', 0)->text();
}
}
$item['id'] = $episode->find('a', 0)->getAttribute('href');
$item['uri'] = self::URI . $htmlepisode->find('.download', 0)->getAttribute('href');
$this->items[] = $item;
}
}
}
public function getName(){
if(!is_null($this->getInput('q'))) {
return $this->getInput('q') . ' : ' . self::NAME;
}
return parent::getName();
}
private function getCachedDate($url){
debugMessage('getting pubdate from url ' . $url . '');
// Initialize cache
$cache = Cache::create('FileCache');
$cache->setPath(CACHE_DIR . '/pages');
$params = [$url];
$cache->setParameters($params);
// Get cachefile timestamp
$time = $cache->getTime();
return ($time !== false ? $time : time());
}
}

View File

@@ -228,6 +228,19 @@ class VkBridge extends BridgeAbstract
$item = array();
$item['content'] = strip_tags(backgroundToImg($post->find('div.wall_text', 0)->innertext), '<br><img>');
$item['content'] .= $content_suffix;
$item['categories'] = array();
// get post hashtags
foreach($post->find('a') as $a) {
$href = $a->getAttribute('href');
$prefix = '/feed?section=search&q=%23';
$innertext = $a->innertext;
if ($href && substr($href, 0, strlen($prefix)) === $prefix) {
$item['categories'][] = urldecode(substr($href, strlen($prefix)));
} else if (substr($innertext, 0, 1) == '#') {
$item['categories'][] = $innertext;
}
}
// get post link
$post_link = $post->find('a.post_link', 0)->getAttribute('href');

View File

@@ -101,7 +101,7 @@ class YGGTorrentBridge extends BridgeAbstract {
. $category
. '&sub_category='
. $subcategory
. '&do=search')
. '&do=search&order=desc&sort=publish_date')
or returnServerError('Unable to query Yggtorrent !');
$count = 0;

View File

@@ -45,9 +45,25 @@ class YoutubeBridge extends BridgeAbstract {
'type' => 'number',
'exampleValue' => 1
)
),
'global' => array(
'duration_min' => array(
'name' => 'min. duration (minutes)',
'type' => 'number',
'title' => 'Minimum duration for the video in minutes',
'exampleValue' => 5
),
'duration_max' => array(
'name' => 'max. duration (minutes)',
'type' => 'number',
'title' => 'Maximum duration for the video in minutes',
'exampleValue' => 10
)
)
);
private $feedName = '';
private function ytBridgeQueryVideoInfo($vid, &$author, &$desc, &$time){
$html = $this->ytGetSimpleHTMLDOM(self::URI . "watch?v=$vid");
@@ -113,6 +129,17 @@ class YoutubeBridge extends BridgeAbstract {
private function ytBridgeParseHtmlListing($html, $element_selector, $title_selector, $add_parsed_items = true) {
$limit = $add_parsed_items ? 10 : INF;
$count = 0;
$duration_min = $this->getInput('duration_min') ?: -1;
$duration_min = $duration_min * 60;
$duration_max = $this->getInput('duration_max') ?: INF;
$duration_max = $duration_max * 60;
if($duration_max < $duration_min) {
returnClientError('Max duration must be greater than min duration!');
}
foreach($html->find($element_selector) as $element) {
if($count < $limit) {
$author = '';
@@ -121,6 +148,20 @@ class YoutubeBridge extends BridgeAbstract {
$vid = str_replace('/watch?v=', '', $element->find('a', 0)->href);
$vid = substr($vid, 0, strpos($vid, '&') ?: strlen($vid));
$title = $this->ytBridgeFixTitle($element->find($title_selector, 0)->plaintext);
// The duration comes in one of the formats:
// hh:mm:ss / mm:ss / m:ss
// 01:03:30 / 15:06 / 1:24
$durationText = trim($element->find('span[class="video-time"]', 0)->plaintext);
$durationText = preg_replace('/([\d]{1,2})\:([\d]{2})/', '00:$1:$2', $durationText);
sscanf($durationText, '%d:%d:%d', $hours, $minutes, $seconds);
$duration = $hours * 3600 + $minutes * 60 + $seconds;
if($duration < $duration_min || $duration > $duration_max) {
continue;
}
if($title != '[Private Video]' && strpos($vid, 'googleads') === false) {
if ($add_parsed_items) {
$this->ytBridgeQueryVideoInfo($vid, $author, $desc, $time);
@@ -168,7 +209,7 @@ class YoutubeBridge extends BridgeAbstract {
}
if(!empty($url_feed) && !empty($url_listing)) {
if($xml = $this->ytGetSimpleHTMLDOM($url_feed)) {
if(!$this->skipFeeds() && $xml = $this->ytGetSimpleHTMLDOM($url_feed)) {
$this->ytBridgeParseXmlFeed($xml);
} elseif($html = $this->ytGetSimpleHTMLDOM($url_listing)) {
$this->ytBridgeParseHtmlListing($html, 'li.channels-content-item', 'h3');
@@ -182,7 +223,7 @@ class YoutubeBridge extends BridgeAbstract {
$html = $this->ytGetSimpleHTMLDOM($url_listing)
or returnServerError("Could not request YouTube. Tried:\n - $url_listing");
$item_count = $this->ytBridgeParseHtmlListing($html, 'tr.pl-video', '.pl-video-title a', false);
if ($item_count <= 15 && ($xml = $this->ytGetSimpleHTMLDOM($url_feed))) {
if ($item_count <= 15 && !$this->skipFeeds() && ($xml = $this->ytGetSimpleHTMLDOM($url_feed))) {
$this->ytBridgeParseXmlFeed($xml);
} else {
$this->ytBridgeParseHtmlListing($html, 'tr.pl-video', '.pl-video-title a');
@@ -215,6 +256,10 @@ class YoutubeBridge extends BridgeAbstract {
}
}
private function skipFeeds() {
return ($this->getInput('duration_min') || $this->getInput('duration_max'));
}
public function getName(){
// Name depends on queriedContext:
switch($this->queriedContext) {

View File

@@ -95,6 +95,7 @@ try {
$whitelist_selection = array_map('strtolower', $whitelist_selection);
}
$showInactive = filter_input(INPUT_GET, 'show_inactive', FILTER_VALIDATE_BOOLEAN);
$action = array_key_exists('action', $params) ? $params['action'] : null;
$bridge = array_key_exists('bridge', $params) ? $params['bridge'] : null;
@@ -180,8 +181,8 @@ try {
header('Content-Type: text/html');
die(buildBridgeException($e, $bridge));
}
die;
} else {
echo BridgeList::create($whitelist_selection, $showInactive);
}
} catch(HttpException $e) {
http_response_code($e->getCode());
@@ -190,81 +191,3 @@ try {
} catch(\Exception $e) {
die($e->getMessage());
}
$formats = Format::searchInformation();
?>
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Rss-bridge" />
<title>RSS-Bridge</title>
<link href="static/style.css" rel="stylesheet">
<script src="static/search.js"></script>
<script src="static/select.js"></script>
<noscript>
<style>
.searchbar {
display: none;
}
</style>
</noscript>
</head>
<body onload="search()">
<?php
$status = '';
if(defined('DEBUG') && DEBUG === true) {
$status .= 'debug mode active';
}
$query = filter_input(INPUT_GET, 'q');
echo <<<EOD
<header>
<h1>RSS-Bridge</h1>
<h2>·Reconnecting the Web·</h2>
<p class="status">{$status}</p>
</header>
<section class="searchbar">
<h3>Search</h3>
<input type="text" name="searchfield"
id="searchfield" placeholder="Enter the bridge you want to search for"
onchange="search()" onkeyup="search()" value="{$query}">
</section>
EOD;
$activeFoundBridgeCount = 0;
$showInactive = filter_input(INPUT_GET, 'show_inactive', FILTER_VALIDATE_BOOLEAN);
$inactiveBridges = '';
$bridgeList = Bridge::listBridges();
foreach($bridgeList as $bridgeName) {
if(Bridge::isWhitelisted($whitelist_selection, strtolower($bridgeName))) {
echo displayBridgeCard($bridgeName, $formats);
$activeFoundBridgeCount++;
} elseif($showInactive) {
// inactive bridges
$inactiveBridges .= displayBridgeCard($bridgeName, $formats, false) . PHP_EOL;
}
}
echo $inactiveBridges;
?>
<section class="footer">
<a href="https://github.com/RSS-Bridge/rss-bridge">RSS-Bridge ~ Public Domain</a><br />
<p class="version"> <?= Configuration::getVersion() ?> </p>
<?= $activeFoundBridgeCount; ?>/<?= count($bridgeList) ?> active bridges. <br />
<?php
if($activeFoundBridgeCount !== count($bridgeList)) {
// FIXME: This should be done in pure CSS
if(!$showInactive)
echo '<a href="?show_inactive=1"><button class="small">Show inactive bridges</button></a><br />';
else
echo '<a href="?show_inactive=0"><button class="small">Hide inactive bridges</button></a><br />';
}
?>
</section>
</body>
</html>

259
lib/BridgeCard.php Normal file
View File

@@ -0,0 +1,259 @@
<?php
final class BridgeCard {
private static function buildFormatButtons($formats) {
$buttons = '';
foreach($formats as $name) {
$buttons .= '<button type="submit" name="format" value="'
. $name
. '">'
. $name
. '</button>'
. PHP_EOL;
}
return $buttons;
}
private static function getFormHeader($bridgeName, $isHttps = false) {
$form = <<<EOD
<form method="GET" action="?">
<input type="hidden" name="action" value="display" />
<input type="hidden" name="bridge" value="{$bridgeName}" />
EOD;
if(!$isHttps) {
$form .= '<div class="secure-warning">Warning :
This bridge is not fetching its content through a secure connection</div>';
}
return $form;
}
private static function getForm($bridgeName,
$formats,
$isActive = false,
$isHttps = false,
$parameterName = '',
$parameters = array()) {
$form = BridgeCard::getFormHeader($bridgeName, $isHttps);
foreach($parameters as $id => $inputEntry) {
if(!isset($inputEntry['exampleValue']))
$inputEntry['exampleValue'] = '';
if(!isset($inputEntry['defaultValue']))
$inputEntry['defaultValue'] = '';
$idArg = 'arg-'
. urlencode($bridgeName)
. '-'
. urlencode($parameterName)
. '-'
. urlencode($id);
$form .= '<label for="'
. $idArg
. '">'
. filter_var($inputEntry['name'], FILTER_SANITIZE_STRING)
. ' : </label>'
. PHP_EOL;
if(!isset($inputEntry['type']) || $inputEntry['type'] === 'text') {
$form .= BridgeCard::getTextInput($inputEntry, $idArg, $id);
} elseif($inputEntry['type'] === 'number') {
$form .= BridgeCard::getNumberInput($inputEntry, $idArg, $id);
} else if($inputEntry['type'] === 'list') {
$form .= BridgeCard::getListInput($inputEntry, $idArg, $id);
} elseif($inputEntry['type'] === 'checkbox') {
$form .= BridgeCard::getCheckboxInput($inputEntry, $idArg, $id);
}
}
if($isActive) {
$form .= BridgeCard::buildFormatButtons($formats);
} else {
$form .= '<span style="font-weight: bold;">Inactive</span>';
}
return $form . '</form>' . PHP_EOL;
}
private static function getInputAttributes($entry) {
$retVal = '';
if(isset($entry['required']) && $entry['required'] === true)
$retVal .= ' required';
if(isset($entry['pattern']))
$retVal .= ' pattern="' . $entry['pattern'] . '"';
if(isset($entry['title']))
$retVal .= ' title="' . filter_var($entry['title'], FILTER_SANITIZE_STRING) . '"';
return $retVal;
}
private static function getTextInput($entry, $id, $name) {
return '<input '
. BridgeCard::getInputAttributes($entry)
. ' id="'
. $id
. '" type="text" value="'
. filter_var($entry['defaultValue'], FILTER_SANITIZE_STRING)
. '" placeholder="'
. filter_var($entry['exampleValue'], FILTER_SANITIZE_STRING)
. '" name="'
. $name
. '" /><br>'
. PHP_EOL;
}
private static function getNumberInput($entry, $id, $name) {
return '<input '
. BridgeCard::getInputAttributes($entry)
. ' id="'
. $id
. '" type="number" value="'
. filter_var($entry['defaultValue'], FILTER_SANITIZE_NUMBER_INT)
. '" placeholder="'
. filter_var($entry['exampleValue'], FILTER_SANITIZE_NUMBER_INT)
. '" name="'
. $name
. '" /><br>'
. PHP_EOL;
}
private static function getListInput($entry, $id, $name) {
$list = '<select '
. BridgeCard::getInputAttributes($entry)
. ' id="'
. $id
. '" name="'
. $name
. '" >';
foreach($entry['values'] as $name => $value) {
if(is_array($value)) {
$list .= '<optgroup label="' . htmlentities($name) . '">';
foreach($value as $subname => $subvalue) {
if($entry['defaultValue'] === $subname
|| $entry['defaultValue'] === $subvalue) {
$list .= '<option value="'
. $subvalue
. '" selected>'
. $subname
. '</option>';
} else {
$list .= '<option value="'
. $subvalue
. '">'
. $subname
. '</option>';
}
}
$list .= '</optgroup>';
} else {
if($entry['defaultValue'] === $name
|| $entry['defaultValue'] === $value) {
$list .= '<option value="'
. $value
. '" selected>'
. $name
. '</option>';
} else {
$list .= '<option value="'
. $value
. '">'
. $name
. '</option>';
}
}
}
$list .= '</select><br>';
return $list;
}
private static function getCheckboxInput($entry, $id, $name) {
return '<input '
. BridgeCard::getInputAttributes($entry)
. ' id="'
. $id
. '" type="checkbox" name="'
. $name
. '" '
. ($entry['defaultValue'] === 'checked' ?: '')
. ' /><br>'
. PHP_EOL;
}
static function displayBridgeCard($bridgeName, $formats, $isActive = true){
$bridge = Bridge::create($bridgeName);
if($bridge == false)
return '';
$isHttps = strpos($bridge->getURI(), 'https') === 0;
$uri = $bridge->getURI();
$name = $bridge->getName();
$description = $bridge->getDescription();
$parameters = $bridge->getParameters();
if(defined('PROXY_URL') && PROXY_BYBRIDGE) {
$parameters['global']['_noproxy'] = array(
'name' => 'Disable proxy (' . ((defined('PROXY_NAME') && PROXY_NAME) ? PROXY_NAME : PROXY_URL) . ')',
'type' => 'checkbox'
);
}
if(CUSTOM_CACHE_TIMEOUT) {
$parameters['global']['_cache_timeout'] = array(
'name' => 'Cache timeout in seconds',
'type' => 'number',
'defaultValue' => $bridge->getCacheTimeout()
);
}
$card = <<<CARD
<section id="bridge-{$bridgeName}" data-ref="{$bridgeName}">
<h2><a href="{$uri}">{$name}</a></h2>
<p class="description">{$description}</p>
<input type="checkbox" class="showmore-box" id="showmore-{$bridgeName}" />
<label class="showmore" for="showmore-{$bridgeName}">Show more</label>
CARD;
// If we don't have any parameter for the bridge, we print a generic form to load it.
if(count($parameters) === 0
|| count($parameters) === 1 && array_key_exists('global', $parameters)) {
$card .= BridgeCard::getForm($bridgeName, $formats, $isActive, $isHttps);
} else {
foreach($parameters as $parameterName => $parameter) {
if(!is_numeric($parameterName) && $parameterName === 'global')
continue;
if(array_key_exists('global', $parameters))
$parameter = array_merge($parameter, $parameters['global']);
if(!is_numeric($parameterName))
$card .= '<h5>' . $parameterName . '</h5>' . PHP_EOL;
$card .= BridgeCard::getForm($bridgeName, $formats, $isActive, $isHttps, $parameterName, $parameter);
}
}
$card .= '<label class="showless" for="showmore-' . $bridgeName . '">Show less</label>';
$card .= '<p class="maintainer">' . $bridge->getMaintainer() . '</p>';
$card .= '</section>';
return $card;
}
}

126
lib/BridgeList.php Normal file
View File

@@ -0,0 +1,126 @@
<?php
final class BridgeList {
private static function getHead() {
return <<<EOD
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="RSS-Bridge" />
<title>RSS-Bridge</title>
<link href="static/style.css" rel="stylesheet">
<script src="static/search.js"></script>
<script src="static/select.js"></script>
<noscript>
<style>
.searchbar {
display: none;
}
</style>
</noscript>
</head>
EOD;
}
private static function getBridges($whitelist, $showInactive, &$totalBridges, &$totalActiveBridges) {
$body = '';
$totalActiveBridges = 0;
$inactiveBridges = '';
$bridgeList = Bridge::listBridges();
$formats = Format::searchInformation();
$totalBridges = count($bridgeList);
foreach($bridgeList as $bridgeName) {
if(Bridge::isWhitelisted($whitelist, strtolower($bridgeName))) {
$body .= BridgeCard::displayBridgeCard($bridgeName, $formats);
$totalActiveBridges++;
} elseif($showInactive) {
// inactive bridges
$inactiveBridges .= BridgeCard::displayBridgeCard($bridgeName, $formats, false) . PHP_EOL;
}
}
$body .= $inactiveBridges;
return $body;
}
private static function getHeader() {
$status = '';
if(defined('DEBUG') && DEBUG === true) {
$status .= 'debug mode active';
}
return <<<EOD
<header>
<h1>RSS-Bridge</h1>
<h2>·Reconnecting the Web·</h2>
<p class="status">{$status}</p>
</header>
EOD;
}
private static function getSearchbar() {
$query = filter_input(INPUT_GET, 'q');
return <<<EOD
<section class="searchbar">
<h3>Search</h3>
<input type="text" name="searchfield"
id="searchfield" placeholder="Enter the bridge you want to search for"
onchange="search()" onkeyup="search()" value="{$query}">
</section>
EOD;
}
private static function getFooter($totalBridges, $totalActiveBridges, $showInactive) {
$version = Configuration::getVersion();
$inactive = '';
if($totalActiveBridges !== $totalBridges) {
if(!$showInactive) {
$inactive = '<a href="?show_inactive=1"><button class="small">Show inactive bridges</button></a><br>';
} else {
$inactive = '<a href="?show_inactive=0"><button class="small">Hide inactive bridges</button></a><br>';
}
}
return <<<EOD
<section class="footer">
<a href="https://github.com/rss-bridge/rss-bridge">RSS-Bridge ~ Public Domain</a><br>
<p class="version">{$version}</p>
{$totalActiveBridges}/{$totalBridges} active bridges.<br>
{$inactive}
</section>
EOD;
}
static function create($whitelist, $showInactive = true) {
$totalBridges = 0;
$totalActiveBridges = 0;
return '<!DOCTYPE html><html lang="en">'
. BridgeList::getHead()
. '<body onload="search()">'
. BridgeList::getHeader()
. BridgeList::getSearchbar()
. BridgeList::getBridges($whitelist, $showInactive, $totalBridges, $totalActiveBridges)
. BridgeList::getFooter($totalBridges, $totalActiveBridges, $showInactive)
. '</body></html>';
}
}

View File

@@ -1,7 +1,7 @@
<?php
class Configuration {
public static $VERSION = '2018-07-17';
public static $VERSION = '2018-08-07';
public static $config = null;

View File

@@ -16,6 +16,8 @@ require __DIR__ . '/FeedExpander.php';
require __DIR__ . '/Cache.php';
require __DIR__ . '/Authentication.php';
require __DIR__ . '/Configuration.php';
require __DIR__ . '/BridgeCard.php';
require __DIR__ . '/BridgeList.php';
require __DIR__ . '/validation.php';
require __DIR__ . '/html.php';
@@ -32,6 +34,17 @@ if(!file_exists($vendorLibSimpleHtmlDom)) {
}
require_once $vendorLibSimpleHtmlDom;
$vendorLibPhpUrlJoin = __DIR__ . PATH_VENDOR . '/php-urljoin/src/urljoin.php';
if(!file_exists($vendorLibPhpUrlJoin)) {
throw new \HttpException('"php-urljoin" library is missing.
Get it from https://github.com/fluffy-critter/php-urljoin and place the script "urljoin.php" in '
. substr(PATH_VENDOR, 4)
. '/php-urljoin/src/',
500);
}
require_once $vendorLibPhpUrlJoin;
/* Example use
require_once __DIR__ . '/lib/RssBridge.php';

View File

@@ -21,15 +21,34 @@ function getContents($url, $header = array(), $opts = array()){
curl_setopt($ch, CURLOPT_PROXY, PROXY_URL);
}
$content = curl_exec($ch);
// We always want the resonse header as part of the data!
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
$curlError = curl_error($ch);
$curlErrno = curl_errno($ch);
curl_close($ch);
if($content === false)
if($data === false)
debugMessage('Cant\'t download ' . $url . ' cUrl error: ' . $curlError . ' (' . $curlErrno . ')');
return $content;
$headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$header = substr($data, 0, $headerSize);
$headers = parseResponseHeader($header);
$finalHeader = end($headers);
if(array_key_exists('http_code', $finalHeader)
&& strpos($finalHeader['http_code'], '200') === false
&& array_key_exists('Server', $finalHeader)
&& strpos($finalHeader['Server'], 'cloudflare') !== false) {
returnServerError(<<< EOD
The server responded with a Cloudflare challenge, which is not supported by RSS-Bridge!<br>
If this error persists longer than a week, please consider opening an issue on GitHub!
EOD
);
}
curl_close($ch);
return substr($data, $headerSize);
}
function getSimpleHTMLDOM($url,
@@ -98,3 +117,38 @@ $defaultSpanText = DEFAULT_SPAN_TEXT){
$defaultBRText,
$defaultSpanText);
}
/**
* Parses the provided response header into an associative array
*
* Based on https://stackoverflow.com/a/18682872
*/
function parseResponseHeader($header) {
$headers = array();
$requests = explode("\r\n\r\n", trim($header));
foreach ($requests as $request) {
$header = array();
foreach (explode("\r\n", $request) as $i => $line) {
if($i === 0) {
$header['http_code'] = $line;
} else {
list ($key, $value) = explode(': ', $line);
$header[$key] = $value;
}
}
$headers[] = $header;
}
return $headers;
}

View File

@@ -1,304 +1,4 @@
<?php
function displayBridgeCard($bridgeName, $formats, $isActive = true){
$getHelperButtonsFormat = function($formats){
$buttons = '';
foreach($formats as $name) {
$buttons .= '<button type="submit" name="format" value="'
. $name
. '">'
. $name
. '</button>'
. PHP_EOL;
}
return $buttons;
};
$getFormHeader = function($bridgeName){
return <<<EOD
<form method="GET" action="?">
<input type="hidden" name="action" value="display" />
<input type="hidden" name="bridge" value="{$bridgeName}" />
EOD;
};
$bridge = Bridge::create($bridgeName);
if($bridge == false)
return '';
$HTTPSWarning = '';
if(strpos($bridge->getURI(), 'https') !== 0) {
$HTTPSWarning = '<div class="secure-warning">Warning :
This bridge is not fetching its content through a secure connection</div>';
}
$name = '<a href="' . $bridge->getURI() . '">' . $bridge->getName() . '</a>';
$description = $bridge->getDescription();
$card = <<<CARD
<section id="bridge-{$bridgeName}" data-ref="{$bridgeName}">
<h2>{$name}</h2>
<p class="description">
{$description}
</p>
<input type="checkbox" class="showmore-box" id="showmore-{$bridgeName}" />
<label class="showmore" for="showmore-{$bridgeName}">Show more</label>
CARD;
// If we don't have any parameter for the bridge, we print a generic form to load it.
if(count($bridge->getParameters()) == 0) {
$card .= $getFormHeader($bridgeName);
$card .= $HTTPSWarning;
if($isActive) {
if(defined('PROXY_URL') && PROXY_BYBRIDGE) {
$idArg = 'arg-'
. urlencode($bridgeName)
. '-'
. urlencode('proxyoff')
. '-'
. urlencode('_noproxy');
$card .= '<input id="'
. $idArg
. '" type="checkbox" name="_noproxy" />'
. PHP_EOL;
$card .= '<label for="'
. $idArg
. '">Disable proxy ('
. ((defined('PROXY_NAME') && PROXY_NAME) ? PROXY_NAME : PROXY_URL)
. ')</label><br />'
. PHP_EOL;
} if(CUSTOM_CACHE_TIMEOUT) {
$idArg = 'arg-'
. urlencode($bridgeName)
. '-'
. urlencode('_cache_timeout');
$card .= '<label for="'
. $idArg
. '">Cache timeout in seconds : </label>'
. PHP_EOL;
$card .= '<input id="'
. $idArg
. '" type="number" value="'
. $bridge->getCacheTimeout()
. '" name="_cache_timeout" /><br />'
. PHP_EOL;
}
$card .= $getHelperButtonsFormat($formats);
} else {
$card .= '<span style="font-weight: bold;">Inactive</span>';
}
$card .= '</form>' . PHP_EOL;
}
$hasGlobalParameter = array_key_exists('global', $bridge->getParameters());
if($hasGlobalParameter) {
$globalParameters = $bridge->getParameters()['global'];
}
foreach($bridge->getParameters() as $parameterName => $parameter) {
if(!is_numeric($parameterName) && $parameterName == 'global')
continue;
if($hasGlobalParameter)
$parameter = array_merge($parameter, $globalParameters);
if(!is_numeric($parameterName))
$card .= '<h5>' . $parameterName . '</h5>' . PHP_EOL;
$card .= $getFormHeader($bridgeName);
$card .= $HTTPSWarning;
foreach($parameter as $id => $inputEntry) {
$additionalInfoString = '';
if(isset($inputEntry['required']) && $inputEntry['required'] === true)
$additionalInfoString .= ' required';
if(isset($inputEntry['pattern']))
$additionalInfoString .= ' pattern="' . $inputEntry['pattern'] . '"';
if(isset($inputEntry['title']))
$additionalInfoString .= ' title="' . $inputEntry['title'] . '"';
if(!isset($inputEntry['exampleValue']))
$inputEntry['exampleValue'] = '';
if(!isset($inputEntry['defaultValue']))
$inputEntry['defaultValue'] = '';
$idArg = 'arg-'
. urlencode($bridgeName)
. '-'
. urlencode($parameterName)
. '-'
. urlencode($id);
$card .= '<label for="'
. $idArg
. '">'
. $inputEntry['name']
. ' : </label>'
. PHP_EOL;
if(!isset($inputEntry['type']) || $inputEntry['type'] == 'text') {
$card .= '<input '
. $additionalInfoString
. ' id="'
. $idArg
. '" type="text" value="'
. $inputEntry['defaultValue']
. '" placeholder="'
. $inputEntry['exampleValue']
. '" name="'
. $id
. '" /><br />'
. PHP_EOL;
} elseif($inputEntry['type'] == 'number') {
$card .= '<input '
. $additionalInfoString
. ' id="'
. $idArg
. '" type="number" value="'
. $inputEntry['defaultValue']
. '" placeholder="'
. $inputEntry['exampleValue']
. '" name="'
. $id
. '" /><br />'
. PHP_EOL;
} else if($inputEntry['type'] == 'list') {
$card .= '<select '
. $additionalInfoString
. ' id="'
. $idArg
. '" name="'
. $id
. '" >';
foreach($inputEntry['values'] as $name => $value) {
if(is_array($value)) {
$card .= '<optgroup label="' . htmlentities($name) . '">';
foreach($value as $subname => $subvalue) {
if($inputEntry['defaultValue'] === $subname
|| $inputEntry['defaultValue'] === $subvalue) {
$card .= '<option value="'
. $subvalue
. '" selected>'
. $subname
. '</option>';
} else {
$card .= '<option value="'
. $subvalue
. '">'
. $subname
. '</option>';
}
}
$card .= '</optgroup>';
} else {
if($inputEntry['defaultValue'] === $name
|| $inputEntry['defaultValue'] === $value) {
$card .= '<option value="'
. $value
. '" selected>'
. $name
. '</option>';
} else {
$card .= '<option value="'
. $value
. '">'
. $name
. '</option>';
}
}
}
$card .= '</select><br >';
} elseif($inputEntry['type'] == 'checkbox') {
if($inputEntry['defaultValue'] === 'checked')
$card .= '<input '
. $additionalInfoString
. ' id="'
. $idArg
. '" type="checkbox" name="'
. $id
. '" checked /><br />'
. PHP_EOL;
else
$card .= '<input '
. $additionalInfoString
. ' id="'
. $idArg
. '" type="checkbox" name="'
. $id
. '" /><br />'
. PHP_EOL;
}
}
if($isActive) {
if(defined('PROXY_URL') && PROXY_BYBRIDGE) {
$idArg = 'arg-'
. urlencode($bridgeName)
. '-'
. urlencode('proxyoff')
. '-'
. urlencode('_noproxy');
$card .= '<input id="'
. $idArg
. '" type="checkbox" name="_noproxy" />'
. PHP_EOL;
$card .= '<label for="'
. $idArg
. '">Disable proxy ('
. ((defined('PROXY_NAME') && PROXY_NAME) ? PROXY_NAME : PROXY_URL)
. ')</label><br />'
. PHP_EOL;
} if(CUSTOM_CACHE_TIMEOUT) {
$idArg = 'arg-'
. urlencode($bridgeName)
. '-'
. urlencode('_cache_timeout');
$card .= '<label for="'
. $idArg
. '">Cache timeout in seconds : </label>'
. PHP_EOL;
$card .= '<input id="'
. $idArg
. '" type="number" value="'
. $bridge->getCacheTimeout()
. '" name="_cache_timeout" /><br />'
. PHP_EOL;
}
$card .= $getHelperButtonsFormat($formats);
} else {
$card .= '<span style="font-weight: bold;">Inactive</span>';
}
$card .= '</form>' . PHP_EOL;
}
$card .= '<label class="showless" for="showmore-' . $bridgeName . '">Show less</label>';
$card .= '<p class="maintainer">' . $bridge->getMaintainer() . '</p>';
$card .= '</section>';
return $card;
}
function sanitize($textToSanitize,
$removedTags = array('script', 'iframe', 'input', 'form'),
$keptAttributes = array('title', 'href', 'src'),
@@ -342,18 +42,11 @@ function backgroundToImg($htmlContent) {
function defaultLinkTo($content, $server){
foreach($content->find('img') as $image) {
if(strpos($image->src, 'http') === false
&& strpos($image->src, '//') === false
&& strpos($image->src, 'data:') === false)
$image->src = $server . $image->src;
$image->src = urljoin($server, $image->src);
}
foreach($content->find('a') as $anchor) {
if(strpos($anchor->href, 'http') === false
&& strpos($anchor->href, '//') === false
&& strpos($anchor->href, '#') !== 0
&& strpos($anchor->href, '?') !== 0)
$anchor->href = $server . $anchor->href;
$anchor->href = urljoin($server, $anchor->href);
}
return $content;

View File

@@ -3,20 +3,53 @@ function search() {
var searchTerm = document.getElementById('searchfield').value;
var searchableElements = document.getElementsByTagName('section');
var regexMatch = new RegExp(searchTerm, "i");
var regexMatch = new RegExp(searchTerm, 'i');
// Attempt to create anchor from search term (will default to 'localhost' on failure)
var searchTermUri = document.createElement('a');
searchTermUri.href = searchTerm;
if(searchTermUri.hostname == 'localhost') {
searchTermUri = null;
} else {
// Ignore "www."
if(searchTermUri.hostname.indexOf('www.') === 0) {
searchTermUri.hostname = searchTermUri.hostname.substr(4);
}
}
for(var i = 0; i < searchableElements.length; i++) {
var textValue = searchableElements[i].getAttribute('data-ref');
if(textValue != null) {
var anchors = searchableElements[i].getElementsByTagName('a');
if(textValue.match(regexMatch) == null && searchableElements[i].style.display != "none") {
if(anchors != null && anchors.length > 0) {
searchableElements[i].style.display = "none";
var uriValue = anchors[0]; // First anchor is bridge URI
} else if(textValue.match(regexMatch) != null) {
// Ignore "www."
if(uriValue.hostname.indexOf('www.') === 0) {
uriValue.hostname = uriValue.hostname.substr(4);
}
searchableElements[i].style.display = "block";
}
if(textValue != null || uriValue != null) {
if(textValue.match(regexMatch) != null ||
uriValue.hostname.match(regexMatch) ||
searchTermUri != null &&
uriValue.hostname != 'localhost' && (
uriValue.href.match(regexMatch) != null ||
uriValue.hostname == searchTermUri.hostname)) {
searchableElements[i].style.display = 'block';
} else {
searchableElements[i].style.display = 'none';
}

131
vendor/php-urljoin/src/urljoin.php vendored Normal file
View File

@@ -0,0 +1,131 @@
<?php
/*
A spiritual port of Python's urlparse.urljoin() function to PHP. Why this isn't in the standard library is anyone's guess.
Author: fluffy, http://beesbuzz.biz/
Latest version at: https://github.com/plaidfluff/php-urljoin
*/
function urljoin($base, $rel) {
if (!$base) {
return $rel;
}
if (!$rel) {
return $base;
}
$uses_relative = array('', 'ftp', 'http', 'gopher', 'nntp', 'imap',
'wais', 'file', 'https', 'shttp', 'mms',
'prospero', 'rtsp', 'rtspu', 'sftp',
'svn', 'svn+ssh', 'ws', 'wss');
$pbase = parse_url($base);
$prel = parse_url($rel);
if (array_key_exists('path', $pbase) && $pbase['path'] === '/') {
unset($pbase['path']);
}
if (isset($prel['scheme'])) {
if ($prel['scheme'] != $pbase['scheme'] || in_array($prel['scheme'], $uses_relative) == false) {
return $rel;
}
}
$merged = array_merge($pbase, $prel);
// Handle relative paths:
// 'path/to/file.ext'
// './path/to/file.ext'
if (array_key_exists('path', $prel) && substr($prel['path'], 0, 1) != '/') {
// Normalize: './path/to/file.ext' => 'path/to/file.ext'
if (substr($prel['path'], 0, 2) === './') {
$prel['path'] = substr($prel['path'], 2);
}
if (array_key_exists('path', $pbase)) {
$dir = preg_replace('@/[^/]*$@', '', $pbase['path']);
$merged['path'] = $dir . '/' . $prel['path'];
} else {
$merged['path'] = '/' . $prel['path'];
}
}
if(array_key_exists('path', $merged)) {
// Get the path components, and remove the initial empty one
$pathParts = explode('/', $merged['path']);
array_shift($pathParts);
$path = [];
$prevPart = '';
foreach ($pathParts as $part) {
if ($part == '..' && count($path) > 0) {
// Cancel out the parent directory (if there's a parent to cancel)
$parent = array_pop($path);
// But if it was also a parent directory, leave it in
if ($parent == '..') {
array_push($path, $parent);
array_push($path, $part);
}
} else if ($prevPart != '' || ($part != '.' && $part != '')) {
// Don't include empty or current-directory components
if ($part == '.') {
$part = '';
}
array_push($path, $part);
}
$prevPart = $part;
}
$merged['path'] = '/' . implode('/', $path);
}
$ret = '';
if (isset($merged['scheme'])) {
$ret .= $merged['scheme'] . ':';
}
if (isset($merged['scheme']) || isset($merged['host'])) {
$ret .= '//';
}
if (isset($prel['host'])) {
$hostSource = $prel;
} else {
$hostSource = $pbase;
}
// username, password, and port are associated with the hostname, not merged
if (isset($hostSource['host'])) {
if (isset($hostSource['user'])) {
$ret .= $hostSource['user'];
if (isset($hostSource['pass'])) {
$ret .= ':' . $hostSource['pass'];
}
$ret .= '@';
}
$ret .= $hostSource['host'];
if (isset($hostSource['port'])) {
$ret .= ':' . $hostSource['port'];
}
}
if (isset($merged['path'])) {
$ret .= $merged['path'];
}
if (isset($prel['query'])) {
$ret .= '?' . $prel['query'];
}
if (isset($prel['fragment'])) {
$ret .= '#' . $prel['fragment'];
}
return $ret;
}