Thanks for taking your time to help me Adze, it's greatly appreciated!

I think you cracked it with "to which address are you trying to connect?".

All this time I was trying to connect to my main static ip pppoe-wan connection which would either time out or give "connection refused".

When I tested nat reflection towards the IP address assigned to the ds-lite (pseudo-wan) interface, it worked just fine.

That's a problem however, the ds-lite address is non globably routable, and only a point to point link for translating IPv6 to IPv4 with carrier grade nat on the other end, so it doesnt help me much, the nat reflection shouldn't happen on that IP.

How would I go about telling mwan3 that my static WAN ip through pppoe is the one I want for nat reflection?

(Last edited by spock on 18 Dec 2014, 11:59)

Would this ip be the "213.16.246.19"? If yes, no extra mwan3 config should be needed... It is not needed for all addresses listed in table mwan3_connected.

As for some more oddity, by a coincidence I tried to connect to the pppoe wan static ip with 'telnet <public-ip> 25', and it worked. Scratching my head I decided to reboot the router and now it doesn't work again.

Can there be something going on with routing/conntrack?

(Last edited by spock on 18 Dec 2014, 12:52)

213.16.246.19 is the bbras of my isp, not my routers public ip.

Right now, when trying to connect to my public wan ip, it just seems to send packets to a black hole, telnet stalls and nothing happen, no errors reported at all, until eventually I get "connection timed out".

(Last edited by spock on 18 Dec 2014, 12:24)

tcpdump running on the router doesn't really reveal anything of interest either. unless you know some better tcpdump-fu I haven't tried yet.

and tcpdump running on the dmz server doesn't even see incoming packets when the client in the main subnet tries to access the public ip for nat reflection. so it stops within the router.

edit: tcpdump running on the dmzserver when the client in the main subnet does 'telnet <wan2(192.0.0.2)> 25' show up normally though.

It's like mwan3 just doesnt want to reflect from the real pppoe-wan IP.

(Last edited by spock on 18 Dec 2014, 13:14)

Is the address which you try to connnect to listed in table mwan3_connected? If not, could you post "ip route" and "iptables -L mwan3_connected -t mangle -v -n"? Thnx!

root@dexter:~# ip route
default dev dslite-wan2  proto static  scope link  metric 20 
default via 213.16.246.19 dev pppoe-wan  proto static  metric 30 
10.xx.xx.0/24 dev br-lan  proto kernel  scope link  src 10.xx.xx.1 
10.xx.xx.0/24 dev eth0.3  proto kernel  scope link  src 10.xx.xx.1 
192.0.0.1 dev dslite-wan2  proto kernel  scope link  src 192.0.0.2 
192.0.0.2 dev dslite-wan2  proto static  scope link  metric 20 
192.168.1.0/24 dev eth0.2  proto kernel  scope link  src 192.168.1.10 
213.16.246.19 dev pppoe-wan  proto kernel  scope link  src 193.92.181.xx

root@dexter:~# iptables -L mwan3_connected -t mangle -v -n
Chain mwan3_connected (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   42  3042 MARK       all  --  *      *       0.0.0.0/0            127.0.0.0/8          MARK or 0xff00
  197 29813 MARK       all  --  *      *       0.0.0.0/0            224.0.0.0/3          MARK or 0xff00
  768  158K MARK       all  --  *      *       0.0.0.0/0            10.xx.xx.0/24        MARK or 0xff00
    9   540 MARK       all  --  *      *       0.0.0.0/0            10.xx.xx.0/24       MARK or 0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            192.0.0.1            MARK or 0xff00
  386 28894 MARK       all  --  *      *       0.0.0.0/0            192.0.0.2            MARK or 0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            192.168.1.0/24       MARK or 0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            213.16.246.19        MARK or 0xff00

The static IP is 193.92.181.xx

Again, thanks for your time!

(Last edited by spock on 18 Dec 2014, 13:20)

Hmm. Pretty strange. There is no entry for 193.92.181.xx in the routing table. Never seen that before.

What does "ip route get 193.92.181.xx" say?

If you were to add it yourself "ip addr add 193.92.181.xx/32 dev pppoe-wan" and then "mwan3 restart". Does this work?

Notice how the first hop bbras IP (213.16.246.19) is listed in mwan3_connected table, instead of my static ip?

root@dexter:~# ip route get 193.92.181.xx
local 193.92.181.xx dev lo  src 193.92.181.xx
    cache <local> 

this: "ip addr add 193.92.181.xx/32 dev pppoe-wan" made no difference to the routing table or functionality after mwan3 restart.

can this all be pppoe related? or perhaps something more general in how openwrt/netifd(?) handles things?

edit: just to clarify I'm not a dumbo here, I'm replacing the identifiable IP's with xx for peace of mind.

(Last edited by spock on 18 Dec 2014, 13:46)

Oh, i can tell..   Never said you were! 

I have to look into this. In the mean time you can use the workaround i posted earlier (create an extra mwan3 rule).

That sounds great, I'll stick with the workaround for now.

Again, many thanks for helping!

edit: I just noticed something curious, a traceroute from router has just started timing out on the first hop:

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 38 byte packets
 1  *  *  *
 2  te0-5-0-2.distr-kln-01.forthnet.gr (194.219.199.113)  13.367 ms  12.171 ms  12.851 ms
 3  core-kln-12Be2.forthnet.gr (213.16.247.13)  20.415 ms  22.871 ms  core-kln-13Be3.forthnet.gr (213.16.247.21)  20.596 ms
 4  core-lsf-08XE-0-0-1.forthnet.gr (212.251.94.18)  41.958 ms  21.857 ms  19.026 ms
 5  74.125.48.74 (74.125.48.74)  23.755 ms  24.430 ms  23.974 ms
 6  72.14.235.45 (72.14.235.45)  23.773 ms  23.859 ms  24.025 ms
 7  google-public-dns-a.google.com (8.8.8.8)  23.753 ms  23.956 ms  24.007 ms

Not sure if it's related. But it's odd never the less.

(Last edited by spock on 18 Dec 2014, 14:16)

After some more tinkering I figured out a second way to fix it:

iptables -A mwan3_connected -t mangle -d aa.bb.cc.dd/32 -j MARK --set-xmark 0xff00/0xff00

After the above command everything works as expected.

I think somewhere the mwan3 scripts fails to pick the real pppoe-wan IP address and instead chooses to use the one of the bbras. This must be pppoe-related dont you think?

root@dexter:~# iptables -L mwan3_connected -t mangle -v
Chain mwan3_connected (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   15  1074 MARK       all  --  any    any     anywhere             127.0.0.0/8          MARK or 0xff00
   89 13677 MARK       all  --  any    any     anywhere             base-address.mcast.net/3  MARK or 0xff00
  394  296K MARK       all  --  any    any     anywhere             10.xx.xx.0/24        MARK or 0xff00
    0     0 MARK       all  --  any    any     anywhere             10.xx.xx.0/24       MARK or 0xff00
    0     0 MARK       all  --  any    any     anywhere             192.0.0.1            MARK or 0xff00
  114  3648 MARK       all  --  any    any     anywhere             192.0.0.2            MARK or 0xff00
    0     0 MARK       all  --  any    any     anywhere             192.168.1.0/24       MARK or 0xff00
    0     0 MARK       all  --  any    any     anywhere             bbras-llu-ath-02L500.forthnet.gr  MARK or 0xff00

(Last edited by spock on 18 Dec 2014, 22:05)

Spent some more time on this today.

I think I figured out why 'mwan3 start' spews out some errors. my wan ipaddr is not listed in uci, I'm guessing due to the fact that it's not a "real" interface but actually pppoe.

I'm guessing mwan3 pulls device-local ip addresses from there?

Look at this:

root@dexter:/etc/config# uci show |grep ipaddr
uci: Entry not found
uci: Entry not found
network.loopback.ipaddr=127.0.0.1
network.lan.ipaddr=10.xx.xx.1
network.modem.ipaddr=192.168.1.10
network.dmz.ipaddr=10.xxx.xx.1
network.loopback.ipaddr=127.0.0.1
network.lan.ipaddr=10.xx.xx.1
network.modem.ipaddr=192.168.1.10
network.dmz.ipaddr=10.xx.xx.1
uci: Entry not found

And since the entries are not "there", we get 'sh: enabled: unknown operand' from the scripts.

root@dexter:/etc/config# uci get network.wan.ipaddr
uci: Entry not found

root@dexter:/etc/config# uci get network.pppoe-wan.ipaddr
uci: Invalid argument

root@dexter:/etc/config# uci get network.modem.ipaddr
192.168.1.10

what do you make of this? are the mwan3 scripts pulling "mwan3_connected" IP's from UCI or am I just confusing myself here?

Now I'm thinking there are of course workarounds to mwan3 script to fix this, but shouldn't this be resolved within ubus/uci?

(Last edited by spock on 19 Dec 2014, 11:49)

I think maybe you can forget my previous guesses..

Around line 90 in /etc/hotplug.d/iface/15-mwan there is a function that pulls the local IP's.

ip route list dev $DEVICE scope link | awk '{print $1}' | egrep '[0-9]{1,3}(\.[0-9]{1,3}){3}'

On a pppoe-wan connection this will pull out the bbras uplink ip, which is why it doesnt work.

So if we first check it it's pppoe-wan (or pseudo interface?), then run this code instead:

ip route list dev $DEVICE scope link | awk '{print $5}' | egrep '[0-9]{1,3}(\.[0-9]{1,3}){3}'

Things should then be working as expected. Even better would be if the ip information could be pulled easier out of ubus, but I have no idea how to do that.

Times like these I wish I was better with bash scripting..

(Last edited by spock on 19 Dec 2014, 13:47)

Hi spock,

Thank you for looking into it, but I already know what's wrong. The above line is correct. It is actually line 70 in the mwan3_set_connected_iptables section which is incomplete. I just haven't had the time to fix it.

The section mwan3_set_iface_iptables is responsible for marking traffic origination from hosts directly connected to wan interface. That it matches bbras ip address is actually correct.

Edit:
The reason that you get the "uci: Entry not found" error is not directly related to mwan3, as you get these errors when you run "uci show" or "uci get". The other enabled error however might be a mwan3 error.

Edit2:
What do you get when you run "uci get -p /var/state network.wan.ifname"?

(Last edited by Adze on 19 Dec 2014, 22:44)

Hi Adaze,

I have a similar problem with the error messages:

mwan3 restart
sh: enabled: unknown operand
uci: Entry not found
uci: Entry not found

Running "uci get -p /var/state network.wan.ifname" returns eth0.4081.

When I run ash -x mwan3 restart, in the output I see:

+ device=eth0.4081
+ [ -n eth0.4081 ]
+ [ 1 -eq 1 ]
+ ACTION=ifup INTERFACE=wan DEVICE=eth0.4081 /sbin/hotplug-call iface
sh: enabled: unknown operand
uci: Entry not found
uci: Entry not found

I don't know if that helps or not.

Hi dadadynobucks,

Thank you. It helps a bit. My hunch that is not mwan3 related, gets stronger and stronger. After the hotplug event is successful triggered, you get the uci errors. At that stage any script in /etc/hotplug.d/iface/ can be responsible for these errors.

(Last edited by Adze on 19 Dec 2014, 23:09)

Hi Adaze,

I removed the scripts in /etc/hotplug.d/iface one at a time. The problem script is 50-miniupnpd. When I removed it, I had no errors.

Nice Thank you!

Thanks for looking it to this more

I also realized that the 'uci show' and 'uci get' errors have nothing to do with mwan3, I just found it odd that uci itself wouldn't list the ifname or ipaddress.

The following on the other hand reports "pppoe-wan" and it's respective ip address as it should.

uci get -p /var/state network.wan.ifname
uci get -p /var/state network.wan.ipaddr

As such I must admit that don't know the underlying code of openwrt/uci/ubus very well yet. :-)

About the following code, doesn't openwrt provide us with any better interface for retrieving such information, since it will break with pppoe, and possible other configurations too(?), just guessing here.

$($IP route | awk '{print $1}' | egrep '[0-9]{1,3}(\.[0-9]{1,3}){3}');

I was looking into ubus, but it quickly became a bit overwhelming, and maybe not suitable.. not to mention it might also need extra packages for parsing(?).

ubus -S call network.interface.wan status

Being christmas vacations and all I found it a good and fun opportunity to do some tinkering in the code and learn more things, to see what I can come up with. So I'll spend some time learning and trying to post a fix/patch for it if I manage.

Thanks for your time and a merry christmas & happy new years!

(Last edited by spock on 20 Dec 2014, 05:59)

I tried to replace the following (which made nat reflection work properly):

~line 70
for connected_networks in $(uci -q show |grep ipaddr | awk -F\= '{print $2}' | tr '\n' ' ' | egrep '[0-9]{1,3}(\.[0-9]{1,3}){3}'); do
              $IPT -A mwan3_connected -d $connected_networks -j MARK --set-xmark 0xff00/0xff00
done

But I started getting timeout and all kinds of havoc was let loose on the network. Strange things.

Try to replace the line

$($IP route | awk '{print $1}' | egrep '[0-9]{1,3}(\.[0-9]{1,3}){3}');

with

$($IP route list table 0 | awk '{print $1}' | egrep '[0-9]{1,3}(\.[0-9]{1,3}){3}');

ip route list table 0

this actually lists my ip, but like this:
local 193.92.xx.xx dev pppoe-wan

So I thought I'd change '{print $1}' to '{print $2}' so it would fetch my IP. But once again local routing got messed up.

Running "iptables -L mwan3_connected -t mangle -v -n" after this change also did not look pretty.

(Last edited by spock on 20 Dec 2014, 11:08)

I think if you add some lines to:

for connected_networks in $($IP route | awk '{print $1}' | egrep '[0-9]{1,3}(\.[0-9]{1,3}){3}'); do
    $IPT -A mwan3_connected -d $connected_networks -j MARK --set-xmark 0xff00/0xff00
done

to

for connected_networks in $($IP route | awk '{print $1}' | egrep '[0-9]{1,3}(\.[0-9]{1,3}){3}'); do
    $IPT -A mwan3_connected -d $connected_networks -j MARK --set-xmark 0xff00/0xff00
done

for connected_networks in $($IP route list table 0 | awk '{print $2}' | egrep '[0-9]{1,3}(\.[0-9]{1,3}){3}'); do
    $IPT -A mwan3_connected -d $connected_networks -j MARK --set-xmark 0xff00/0xff00
done

You will be ok to go.

Edit: I don't know why the second do in the last paste is on a new line?? I did not post it that way..

(Last edited by Adze on 20 Dec 2014, 11:15)

You're right, that fixed it!

Many many thanks for your time.

(Last edited by spock on 20 Dec 2014, 11:19)