dr|z3d
orignal: you're in someone's sights :) teddit.i2p/r/i2p/comments/zp8j2j/i2pd_developer_trusts_the_russian_government
Xeha
lol
Xeha
they cant even spell his name correctly
dr|z3d
INFO […PReader 1/6] …rkDatabaseFacade: Peer [zQXMHP] published their RouterInfo in the future
dr|z3d
• Publish date: Tue Jul 29 22:57:31 GMT 2031
dr|z3d
Peer [p6Kv8M] published their RouterInfo in the future
dr|z3d
• Publish date: Sun May 16 14:52:42 GMT 2038
dr|z3d
Peer [1Q2Xxi] published their RouterInfo in the future
dr|z3d
• Publish date: Thu Dec 30 22:43:48 GMT 2100
dr|z3d
Peer [hGXAhz] published their RouterInfo in the future
dr|z3d
• Publish date: Fri Oct 20 18:26:42 GMT 2023
dr|z3d
Peer [9jCtNr] published their RouterInfo in the future
dr|z3d
• Publish date: Sat Jun 28 21:06:05 GMT 2025
dr|z3d
Peer [a1W29t] published their RouterInfo in the future
dr|z3d
• Publish date: Sun Apr 23 00:24:02 GMT 2023
dr|z3d
Peer [ZQ0LLc] published their RouterInfo in the future
dr|z3d
• Publish date: Mon Feb 01 07:39:29 GMT 2027
dr|z3d
just a hunch, but this doesn't look accidental.
zzz
clock fixes coming today because I got blown up twice in a week
zzz
orignal, yes, I am concerned about clock bias
zzz
please change to "round"
zzz
please check and change NTCP2 also
zzz
thank you
zzz
dr|z3d, LFnvaZ8Qa-0IRiz55PsQlk6Bw3EZ6J5tHsIv5UHtdtE=
dr|z3d
what oddball behavior you seeing on that one, zzz?
zzz
some hop throttles but maybe false alarm
zzz
build success just hit lowest in a decade at least
dr|z3d
given what I've been seeing lately, I have a feeling that's by design.
dr|z3d
either that or a whole bunch of seeminly unrelated bugs are surfacing at the same time which seems unlikely.
zzz
ok holler if you see any new culprits
dr|z3d
what did you make of those future routers?
dr|z3d
all within a couple of minutes of each other.
zzz
nothing new, and haven't seen very often, hard to say
zzz
add'l defenses checked in
dr|z3d
yeah, same, not something I've seen much, but 2100!!! that's beyond clockskew issues.
dr|z3d
speaking of the addl defences, super minor logging typo you got going on in a couple of places. Stratrum -> Stratum.
zzz
ok thx
dr|z3d
np. minor, but noticed as I'm in the process of manual mergery.
zzz
LFnv looks like a healthy X router
zzz
attacker can easily drive a X router to get caught in other people's throttles
zzz
which is why I think it's a bad idea to ban for hop throttle, at least not for more than a few minutes
zzz
otherwise attacker could drive whole network to collapse
dr|z3d
yeah, a few minutes is what I'm currently banning for.
zzz
ok
dr|z3d
and given my throttle's a bit more lax than yours, not much gets caught in it, though I've seen some stuff snared since I tightened it.
dr|z3d
another future router, this one from 2027.
zzz
anything in common among these?
dr|z3d
not that I can tell. I'm trying to get them in the short-term sin bin so I don't have to constantly monitor the logs.
zzz
i2pd? java?
zzz
I don't think java will go more than 25 years out
zzz
ref: BuildTime
dr|z3d
I don't know, I don't think I'm snagging the routerid the correct way, so searches are coming up blank. and temp banning also failing.
dr|z3d
String riHash = routerInfo.getIdentity().getHash().toBase64().substring(0,6);
dr|z3d
if (routerInfo.getPublished() > now + 2*Router.CLOCK_FUDGE_FACTOR) {
dr|z3d
long age = routerInfo.getPublished() - now;
dr|z3d
if (_log.shouldWarn()) {
dr|z3d
_log.warn("Peer [" + riHash + "] published their RouterInfo in the future\n* Publish date: " + new Date(routerInfo.getPublished()));
dr|z3d
_log.warn("Banning [" + riHash + "] for 15m -> RouterInfo from the future!");
dr|z3d
}
dr|z3d
_context.banlist().banlistRouter(routerInfo.getIdentity().getHash(), " <b>➜</b> RouterInfo from the future (" +
dr|z3d
new Date(routerInfo.getPublished()) + ")", null, null, 15*60*1000);
zzz
yeah you can't search for them
dr|z3d
ah, yeah, course, they don't make it to the netdb.
zzz
you'd have to log the full RI just before the ban
zzz
I believe we do in SSU2
dr|z3d
so looking at the above, I guess it's fail because I'm trying to ban using a non-existent routerinfo as the reference.
zzz
no, you ban by hash, that works
dr|z3d
yeah, except it's not working for some reason. nothing turning up in the banlist :|
dr|z3d
the only thing at this stage I've noticed is that sometimes you'll see 1/2 dozen routers with future routerinfos all turn up at more or less the same time.
zzz
you;ll see the full RI for SSU2 with EstablishmentManager at WARN
zzz
ofc if you are ff then you'll get a lot more of them as database stores
zzz
transports will usually catch skew before it gets to netdb
dr|z3d
yeah, so I'm thinking that's probably why the call to routerinfo.getPublished().getHash() isn't working because there's no RI to pull the hash from.
zzz
if routerInfo were null you'd NPE
dr|z3d
hmm, yeah, I guess.
dr|z3d
so idk why I'm not seeing bans. oh well.
dr|z3d
anyways, I'm suggesting short term bans for anything that looks like it might be hostile. 5-10m. excessive hop requests, hostile requests, future routerinfos.
dr|z3d
it'll at least place the RI's in the banlist so they can be investigated, without doing too much harm given the brevity of the ban.
dr|z3d
s/RI's/hashes
dr|z3d
pv44hG seems to be a recurring futures merchant.
dr|z3d
(2030)
orignal
dr|z3d the source is kislitsa.i2p ))
dr|z3d
of that particular future routerinfo, orignal?
orignal
zzz, but I don't understand why should I publish clock from future?
orignal
about that page
dr|z3d
oh, you mean orignal is working for the FSB? that one?
orignal
it's well known )))
orignal
but really back to timestamp from future. why?
dr|z3d
for routerinfos? either misconfiguration malicious intent?
dr|z3d
*or
orignal
no, it's zzz change
orignal
because he rounds timestamps instead truncate
dr|z3d
oh, we're not talking about routerinfos from 2100 then? ok, I'll butt out.
zzz
orignal, it's max 500 ms in the future
zzz
the potential problem is when we adjust our local clock to the median of all the peers
orignal
dr|z3d kislitsa.i2p/i2p/59642
orignal
the original topic ))
zzz
if the median is biased, then the network could gradually adjust down down down
orignal
zzz but why I should even round it
zzz
to prevent bias
orignal
so where do you want to to do?
orignal
timestamp block for NTCP2 and SSU2?
zzz
send ms + 500 / 1000 in both SSU2 and NTCP2 DateTime block, yes
orignal
zzz another question
orignal
when you connect through NTCP2 do you validate your own RI?
orignal
if timestamp is not too old
zzz
looking...
orignal
fine will do
zzz
thank you very much :)
orignal
people complain about errors in SessionCorimed that RI is too old
orignal
hmm for i2pd I don't truncate anything
orignal
I take seconds from system clock
orignal
*for NTCP2
zzz
The max time before we rebuild our RI is 43 minutes
orignal
I check for 90 minutes
orignal
and get RIs older in SessinConfirmed
zzz
i2pd or java?
orignal
not sure
orignal
people see it in the logs
zzz
yeah, it's strange, because I reject skew in session or token request and send termination
orignal
now skew
zzz
I don't know how you send a RI that's way too old or in the future in session confirmed but you had the right time in session request
orignal
clock is fine
orignal
RI is old
zzz
right
orignal
e.g. timestamp in it
zzz
I just looked through my logs, all the routers I see with times in the future are i2pd
orignal
how can you explain it?
zzz
no ideas...
orignal
for how long in future?
zzz
dr|z3d reported year 2100 above
orignal
in which message?
dr|z3d
2100, 2023, 2038.. all over the shop.
zzz
I think the most I've seen in SSU2 is 91 days in the future, but I'm not logging the same things dr|z3d is, I'm not floodfill
zzz
he's seeing 100x the traffic I see
orignal
in SessionRequest or SessionCreated?
orignal
so you want me take millisecnds and round to seconds for NTCP2?
zzz
yes please, same as SSU2
zzz
zgrep -i 'in the future' log*|wc -l
zzz
171
zzz
that covers about a week
zzz
mostly session created but some relay too
zzz
I think session/token request is a different log line
orignal
commited
zzz
thx :)
zzz
there should be very few java routers with bad clocks because we have NTP and also as backup take time from peer median, DoH headers, and reseed HTTP headers
zzz
I did fix a few bugs today for some corner cases, but those only happen if NTP fails
zzz
I found those by testing with NTP and reseed and NTCP2 and SSU1 _all_ disabled
orignal
we don't use NTP by deafult
dr|z3d
mostly not seeing the issues on floodfill routers, zzz, or at all.
zzz
interesting
dr|z3d
to clarify, not monitoring floodfills, the logs all coming from non-ff routers. not that ff routers don't have those issues, just haven't been looking at the ffs.
orignal
we see to much transit in last few days
orignal
sometimes I hit 100 Mbs cap of the network port
dr|z3d
yeah, that'll be what we're calling the tunnel spammers.
orignal
how do you know if it's not real traffic?
dr|z3d
depends what you mean by real. 42MB/s is definitely real.
dr|z3d
what's also real is the fact that the requests seem to be originating from 1 or 2 routers.
orignal
where?
orignal
I see traffic not requests
dr|z3d
before we identified the culprits and blocked them, I was seeing 42MB/s on one of the stormycloud outproxy routers.
dr|z3d
all transit, not op traffic.
dr|z3d
and also was watching part tunnels scale from, say 5K to 12K in a few minutes. not normal. tunnel spammers.
dr|z3d
I suspect 42MB/s would have been much higher were it not for the fact that ~40MB/s seems to be about as much as a quadcore VPS can handle.
dr|z3d
yeah, as I was saying, orignal, it's definitely 1 or 2 routers generating a huge amount of traffic and tunnel requests. either maliciously, or just cluelessly using some crufty BOB-enabled retroshare.
orignal
it doesn't expain where thetraffic come from
dr|z3d
and it's likely they're have a negative effect on the whole network.
orignal
I see traffic not requests
orignal
Transit Tunnels: 5776
orignal
not too many
dr|z3d
you running with default i2pd configs?
orignal
ofc not
dr|z3d
ok, just checking :)
dr|z3d
almost 6K tunnels I think you'd find is above average.
orignal
not too much for a floodfill
dr|z3d
but anyways, just because your particular router's not seeing a huge number of requests doesn't mean they're not out there fucking up the network. which they are :)
dr|z3d
tunnel build success network-wide is way down, zzz is reporting.
dr|z3d
and it's not persistent, the tunnel request hike.. it spikes for a few minutes and then it's gone until the next one. the traffic spikes could last anywhere up to 8-12 hours at a time.
zzz
0) Hi
zzz
hi
eyedeekay
hi
zzz
what's on the agenda for today?
zzz
I have two:
orignal
hi
zzz
1) peer test with firewalled charlie, continued from last week
orignal
how you handle peer test at Bob
zzz
2) peer test proposal, didn't have time last week
orignal
from last meeting
zzz
is that the same as 1) or something different?
orignal
different
orignal
1 and 2 are different
zzz
ok, let's make 'how you handle peer test at bob' be 1a)
orignal
yes
orignal
you promised to tell
zzz
eyedeekay, you have anything for the list?
zzz
can always add at the end too
eyedeekay
A very small go-i2p update
zzz
ok you will be 3)
zzz
1) peer test with firewalled charlie, continued from last week
orignal
yes, so what'syou conculsion?
zzz
I promised to do research
zzz
so I did
zzz
as a reminder, orignal allows firewalled charlies, I currently do not
zzz
my conclusion is that I can make the changes so I can handle it
orignal
yes
orignal
nice
dr|z3d
can we make 4) your observations re network performance, zzz. just to put orignal in the picture?
zzz
so I now agree with orignal and I'm planning to make those changes
zzz
anything else on 1) or can we move on to 1a) ?
orignal
no
zzz
1a) how you handle peer test as bob
orignal
so my question is
orignal
when Bob gets error code from charlie what do you do?
zzz
until a week ago, I just forwarded the error to alice. Last week I checked in changes, Bob now asks a different charlie
zzz
up to a max time or max number of charlies.
orignal
yes I do the same
zzz
did that answer your question?
orignal
how many chrlies? 2?
zzz
maybe 5 or so?
orignal
idk
orignal
I'm going to do the same
zzz
ok. anything more you need to know?
orignal
becuase too many errors back came from Charie
orignal
no
orignal
just to confirm
zzz
ok
zzz
on to 2) peer test proposal
zzz
it's here which you probably have not read: zzz.i2p/topics/3489-proposal-ssu2-snat-test-improvement
zzz
it's about how you find out if you are SNAT
zzz
the SSU 1 spec says, if you don't get message 5, then stop the test
zzz
but if you don't send message 6, you'll never find out if you are SNAT
orignal
right
orignal
you do
orignal
when you connect to Bob
zzz
so, proposal is wait a while, then send message 6 anyway
zzz
for SSU2
zzz
and also to fix up the documentation to make it more clear, because the SSU 1 docs are a mess, and we didn't copy that section to SSU 2 at all
orignal
please explain how come I connect to Bob and don't know I'm SNAT?
orignal
for SSU2
orignal
Bob send address block in SessionCreated
zzz
true, you can get the information in other ways
zzz
but peer test would be better. Right now you can only be SNAT if you are not FIREWALLED from peer test.
orignal
sending 6 could be a good idea only you receive 4
zzz
which makes no sense
orignal
huh?
orignal
for me no difference
zzz
SSU2 is better because msg 5 and msg 7 are different. In SSU 1 you can't tell the difference
orignal
if I get different port in address block it means SNAT
zzz
so you can figure out if you got msg 5 after sending msg 6
zzz
anyway, not asking for any decisions today, please read the zzz.i2p post and we can talk about it more some other time
orignal
fine
zzz
anything else on 2) ?
orignal
no
zzz
3) a very small go-i2p update
zzz
you're up eyedeekay
eyedeekay
Yeah sorry having trouble with lag today
eyedeekay
Only real update is that I've finally got what I think is a working Noise-XK handshake and can send and recieve/encrypt and decrypt messages
zzz
woot, congrats
zzz
next steps?
eyedeekay
I was pretty close to it last week as it turned out but finally got my head around the last few things this week
eyedeekay
I need to figure out how to make the NTCP transport inherit from the Noise transport, and then add the ability to apply our mods to the handshake
zzz
sounds like fun
zzz
anything else on 3) ?
eyedeekay
Nope, that's it for me
zzz
4) network performance
zzz
I wasn't prepared to give a presentation, but if anybody has any questions I'll try to answer them, or dr|z3d this is your topic, if you want to lead
zzz
last call for 4)
zzz
we agreed last week that we're off on the 26th and the 2nd
orignal
so
dr|z3d
you want me to start, zzz, or you've got it covered?
orignal
yes please
zzz
somebody go, not my topic
dr|z3d
sure, I can say a few words by way of observations.
dr|z3d
essentially we've been seeing some fairly atypical behavior on the network lately.
dr|z3d
regular spikes in tunnel requests, huge sustained hikes in bandwidth usage over several hours.
dr|z3d
additionally, some strange behavior has been observed including future routerinfos from 2100, requests that place us two hops in the tunnel chain, and various other stuff.
dr|z3d
am I lagged?
orignal
no
dr|z3d
ok, good. maybe zzz is then.
orignal
I see abnormal transit traffic but that's all
orignal
no major issues
dr|z3d
so, it's hard to know at this stage if the behavior is part of some sustained attack and an attempt to probe for weaknesses, or just some incompetents running retorshare and BOB and absolutely hammering the network.
orignal
or more traffic
zzz
orignal, one router was appearing at 20 different IPs per hour, all over the globe
dr|z3d
or it could be some latent bugs are now being exposed, maybe due to SSU2.
zzz
that router I'd call an attack
orignal
but what's thier goal?
zzz
doesn't matter and impossible to know
zzz
even classifying attack vs. user error is a waste of time mostly
orignal
not realy
orignal
they they used kinds "p2p vpn"
dr|z3d
like I mentioned earlier, with a couple of routers blocked, the traffic on the router's I'm watching has returned to what one might describe as normal bandwidth patterns.
zzz
we're monitoring the situation closely. If things get bad we could do a release on short notice. Hopefully not necessary
zzz
we're currently at the worst exploratory build success in maybe 10 years
dr|z3d
if the goal is to destabilize the network and frustrate users, then perhaps it's having some effect.
dr|z3d
but we can only speculate.
dr|z3d
is it a continuation of the attack on Tor by other means? also possible.
dr|z3d
it's curious that one of stormycloud's outproxy routers got hit hard, repeatedly.
dr|z3d
massive traffic (up to 42MB/s).. restart router, almost instantly traffic resumes at previous levels.
zzz
it's some combination of attacks and SSU2 bugs
orignal
yes I see low success rate
zzz
if we go down below 20% expl. success (sustained) I'll probably call for a release
dr|z3d
question, zzz. is there anything you think i2pd might be able to do to mitigate this that it's not doing?
orignal
when?
zzz
I'd rather not do a release before mid-january.
zzz
but if we have to I will. Don't want to wait until everything breaks
zzz
dr|z3d, I'm not in a position to offer i2pd design advice
zzz
other than fix all the SSU2 bugs soon :)
orignal
which bugs?
zzz
anything else on 4) ?
zzz
none in particular
zzz
just anything that keeps it from working reliably
dr|z3d
ok, well, no design advice from zzz, then orignal. :)
dr|z3d
I think we covered the network perf issues. I'll keep reporting any anomalies.
zzz
anything else for the meeting?
orignal
no
zzz
happy holidays, thanks everybody
dr|z3d
happy crystal methmas to all!
zzz
if we do both decide to release, we need to coordinate on whether to go from 0.9.56 to .57 or not, so stay in touch
orignal
fine
dr|z3d
the other thing to keep an eye on, orignal, is known peers in your netdb. that's also seeing a considerable ramp up on occasion. like now.
zzz
orignal, can you please stop i2pd from publishing a "::1" IP? 2.7% of the routers (well over 100) in my netdb are publishing ::1
zzz
I do validate the IP later but it still screws me up some
dr|z3d
this is where ranges and masks for ipv6 would come in handy :)
dr|z3d
eg: <a href="https://freenetworktutorials.com/ipv4-and-ipv6-bogon-address-list/" target="_blank">Bogon</a>:;;1/128
zzz
no. as I said, we validate it later, but that's more expensive
dr|z3d
how do you validate localhost? are you assuming later the router will have self-corrected and is publishing a public ip?
zzz
well, invalidate
zzz
via TranportUtil.isPubliclyRoutable()
dr|z3d
ah, ok.
zzz
orignal, if your IP is ::1 please don't publish a IPv6 address at all, for either NTCP2 or SSU2
dr|z3d
but that's expensive, you're saying? so there's no value in a default ipv6 bogon blocklist to pre-empt that?
zzz
the method mentioned above _is_ our validity check for ipv6
zzz
the point is that comes after cheap checks for a ipv6 address
zzz
please don't undercut me when I'm asking i2pd to fix something
dr|z3d
sorry, wasn't intending to undercut you. not my intention. was just trying to get a handle on why the cheapest check of all, namely blocklist masks/ranges, are the first port of call for blocking private ipv6.
dr|z3d
*aren't.
dr|z3d
orignal: ::1 is calling and it's wearing a grunch suit!
dr|z3d
*grinch
dr|z3d
that's my attempt at humor, and no, I'm not associating you with the grinch, zzz. :)
zzz
you wanted me to make suggestions to i2pd, so here's one, and now you're saying I'm wrong
dr|z3d
no, I'm not saying you're wrong at all.
dr|z3d
I think preventing i2pd from publishing private addresses is entirely reasonable.
dr|z3d
That doesn't mean you can't *also* look at other strategies to mitigate what's apparently a pain point. orignal fixing the issue in i2pd will still leave older routers out there.
dr|z3d
and supporting masks/ranges in ipv6 isn't just about localhost, either. no doubt in time we'll encounter entire ipv6 ranges that may be desirable to block en masse.