OpenWrt Forum Archive

Topic: RSS downloader

The content of this topic has been archived between 29 Mar 2018 and 4 May 2018. Unfortunately there are posts – most likely complete pages – missing.

you need to use leech-defmission recipe.

stangri wrote:

you need to use leech-defmission recipe.

doesnt work neither

transmission recipe? have you configured either defmission or transmission recipe?

tried both and default recipe.
as i understand default recipe must just download file to directory, without adding it to transmission, but it doesnt do it

leech 1.5-2 is out. It's a bugfix release mostly focused on leech for Ubuntu (and probably Debian), it fixes $HISTORY feature on those platforms. $HISTORY on OpenWRT wasn't affected. It also fixes leech-match-test for all platforms including OpenWRT, but if you're not concerned with that issue then update is not necessary.

Changelog:

1.5-2

[*] fixed RFC822 timestamp parsing issue on Ubuntu (and apparently Debian)
[*] fixed similar issue (grep options) in leech-defmission recipe
[*] fixed issue in leech-match-test when sometimes it wasn't reporting file
    as matched pattern(s) when it did (if filename had multiple whitespaces
    in it)

Downloads: https://bitbucket.org/alekseyt/leech/downloads

(Last edited by aleksey_t on 17 Jan 2016, 11:29)

@Udavf
Sorry for the late reply, for some reason i didn't get notifications about replies in this thread...

If it doesn't show any errors and still download anything, then i would say download filters in CONFIG_DIR/wild-downloads and CONFIG_DIR/downloads are not set correctly.

Simple way to check this is to set .* (dot asterisk) filter in CONFIG_DIR/downloads - that should download everything. If it still doesn't, then normally i'm asking for entire content of CONFIG_DIR preferably emailed to me to check what might be wrong.

As a side note, you can always email me directly (email on BitBucket page).

(Last edited by aleksey_t on 17 Jan 2016, 11:30)

@aleksey_t
Thanks for the updates! I noticed that if $COOKIE is not set, the '-b' flag is immediately followed by $FOOD, leading to the following error:
Downloading feed: http://..../ Failed: 4

Is it possible to make the '-b' flag optional until $COOKIE is found?

@discrucio
Thank you for pointing that out. I wasn't able to reproduce this issue, but making -b optional is probably the correct approach anyway. I'll get to it shortly.

discrucio wrote:

@aleksey_t

Many thanks for the great script. I would like to share my recipe for aria2. Uses aria2's JSON-RPC interface. Tested with leech 1.2-1 entware package on OpenWRT BB.

ARIA2_SERVER="http://"$HOST_PORT"/jsonrpc"

ARIA2_PARAM='{"id":null,"jsonrpc2":"2.0","method":"aria2.addUri","params":["token:'"$RPC_SECRET"'",["'$LEECH_URL'"]]}'

exec curl -X POST \
-H 'Content-Type: application/json' -H 'Accept: application/json' \
-d $ARIA2_PARAM $ARIA2_SERVER

I'm struggling to add an option of  $DOWNLOADS_DIR to the rpc call to aria2. Can anyone help me please? I'm not sure if that's bash or json or something but I've tried this:

ARIA2_PARAM='{"id":null,"jsonrpc2":"2.0","method":"aria2.addUri","params":["token:'"$RPC_SECRET"'",["'$LEECH_URL'"],["dir:'"$DOWNLOADS_DIR"'"]]}'

and it didn't work, here's what I've got in the log:

2016-04-17 20:06:48.419203 [INFO] [rpc_helper.cc:103] Executing RPC method aria2.addUri
2016-04-17 20:06:48.419459 [DEBUG] [RpcMethod.cc:101] Exception caught
Exception: [RpcMethodImpl.h:68] errorCode=1 The parameter at 1 has wrong type.
2016-04-17 20:06:48.419505 [DEBUG] [HttpServer.cc:296] HTTP Server sends response:
HTTP/1.1 400 Bad Request

I've tried to read up on arrays and structures in json but couldn't figure out what's wrong with my request.

(Last edited by stangri on 18 Apr 2016, 04:09)

@discrucio
I've made changes in 33fccd9 (https://bitbucket.org/alekseyt/leech/co … c262e68dce), leech won't set -b option unless $COOKIE is set. Hopefully those changes would resolve your difficulty with cookies. I would appreciate if you tell me if it does, if it doesn't i can look into this issue further.

@aleksey_t
Works great, thanks!

@stangri
Try:

ARIA2_PARAM='{"id":null,"jsonrpc":"2.0","method":"aria2.addUri","params":["token:'"$RPC_SECRET"'",["'$LEECH_URL'"],{"dir":"'$LEECH_DOWNLOADS_DIR'"}]}'

I will do some more tests and submit a PR if it works

discrucio wrote:

@aleksey_t
Works great, thanks!

@stangri
Try:

ARIA2_PARAM='{"id":null,"jsonrpc":"2.0","method":"aria2.addUri","params":["token:'"$RPC_SECRET"'",["'$LEECH_URL'"],{"dir":"'$LEECH_DOWNLOADS_DIR'"}]}'

I will do some more tests and submit a PR if it works

Well, I set DOWNLOADS_DIR and not LEECH_DOWNLOADS_DIR, but other than that the line above works! Thank you so much, you wouldn't believe how much time I spent trying to get the syntax right on this one! wink

@discrucio
Note that lately were was TARGET_DIR implemented for leech-transmission recipe. The idea behind this variable is that Transmission downloads dir could be overridden with variable set in environment, like TARGET_DIR=/my/dir CONFIG_DIR=/etc/leech /usr/sbin/leech and this would override any downloads dir configured in Transmission.

The point is that files downloaded by leech are not the same files that downloaded by Transmission and destination directories could be different.

I don't know if aria2 has own downloads directory for files and if it would overlap with leech's downloads dir, and if that matters at all. I can take whatever changes you're happy with, just wanted to point that out, maybe you would find this idea appealing for aria2 recipe.

I'm glad to announce that leech 1.6 is out. Many thanks to stangri and discrucio.

Changelog:

1.6

[+] leech now support TARGET_DIR option in leech-transmission and leech-aria2
    recipes to explicitely set downloads dir in external downloader software
    (Transmission and aria2 respectively) (by stangri, /pull-request/3/)
[+] leech will pass LEECH_TARGET_DIR to each invoked recipe (for recipes that
    could use this option)

[*] fixed issue when leech was setting empty -b option to cURL when $COOKIE
    wasn't set in environment

(Last edited by aleksey_t on 21 Jun 2016, 12:24)

Credit should go to discrucio for providing the proper json syntax for aria2. wink

@aleksey_t

I tried installing the latest package in LEDE snapshot r3582 and got the following error:

Collected errors:
 * pkg_init_from_file: Malformed package file leech_1.6-1_all.ipk.

I wrote a quick Makefile to re-package your scripts using LEDE buildroot and the ipk installed successfully.

If you permit it, I will submit a PR to the Packages repository in Github.

@discrucio
[Disregard edited out text]. I've got what you want to do. You can do whatever, it's a free software.

(Last edited by aleksey_t on 2 Mar 2017, 23:32)

discrucio wrote:

@aleksey_t

I tried installing the latest package in LEDE snapshot r3582 and got the following error:

Collected errors:
 * pkg_init_from_file: Malformed package file leech_1.6-1_all.ipk.

I wrote a quick Makefile to re-package your scripts using LEDE buildroot and the ipk installed successfully.

If you permit it, I will submit a PR to the Packages repository in Github.

Hi

Can You share re-package because got problem with install ipk on LEDE....

@frunzie

Hi there, I didn't submit the PR because the contribution guidelines require a package license (I couldn't find any in aleksey_t's repo). Here's the makefile I used:

#
# Copyright (C) 2004-2018 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#

include $(TOPDIR)/rules.mk

PKG_NAME:=leech
PKG_VERSION:=1.6-2
PKG_RELEASE:=1

PKG_SOURCE_PROTO:=git
PKG_SOURCE_URL:=https://bitbucket.org/alekseyt/leech.git
PKG_SOURCE_VERSION:=749cd75f242024a9d39d7671e95d7b40e2f35e99
PKG_MIRROR_HASH:=b89c5d0846cf9d0e8736754e47a50538a80a7876886ad0737c1a30d7960f3e76

PKG_INSTALL:=1

include $(INCLUDE_DIR)/package.mk

define Package/leech
    SECTION:=net
    CATEGORY:=Network
    SUBMENU:=Download Manager
    TITLE:=RSS-feeds parser/files downloader
    URL:=https://bitbucket.org/alekseyt/leech/
    DEPENDS:=+xsltproc +curl
endef

define Package/leech/description
    leech downloads files for you if you give it RSS-feeds of your favorite trackers.
endef

define Build/Compile
endef

define Build/Install
endef

define Package/leech/install
    $(INSTALL_DIR) $(1)/usr/sbin
    $(INSTALL_BIN) $(PKG_BUILD_DIR)/sbin/* $(1)/usr/sbin

    $(INSTALL_DIR) $(1)/etc/leech
    $(INSTALL_CONF) $(PKG_BUILD_DIR)/config/* $(1)/etc/leech

    $(INSTALL_DIR) $(1)/usr/share/leech
    $(INSTALL_DATA) $(PKG_BUILD_DIR)/share/leech/* $(1)/usr/share/leech
endef

define Package/leech/conffiles
/etc/leech/default
/etc/leech/downloads
/etc/leech/foods
/etc/leech/wild-downloads
/etc/leech/reverse-downloads
endef

$(eval $(call BuildPackage,leech))

The discussion might have continued from here.