IRCaBot 2.1.0
GPLv3 © acetone, 2021-2022
#ls2
/2023/02/05
weko Stupid _
weko [10:26:56] <weko_> Fake floodfill spamming
weko [10:27:03] <weko_> Floodfills: 7717
weko [10:32:50] <weko_> zzz: dr|z3d have you floodfill profiling?
weko [10:33:29] <weko_> Floodfills: 7454
weko [10:33:29] <weko_> Normal value is 1700-1900
weko [11:01:08] <weko_> Check stats.i2p...
weko [10:15:01] <weko_> Attack on the network
weko [10:15:04] <weko_> Very big
weko [10:15:14] <weko_> Tunnel creation success rate: 5%
weko [10:15:14] <weko_> Total tunnel creation success rate: 30%
dr|z3d now that you mention it, weko, I see a fairly huge spike in ff's.
weko For now attack eneded
weko Ended
weko Floodfills: 1787
dr|z3d around twice that here.
dr|z3d let's see if there's anything shared.
weko Previous attack and current attack?
weko It is maybe tor attacker. Vort said it is similar to attack for tor, what attacker changing behavior and have some pauses
weko dr|z3d: have java i2p floodfill profiling?
dr|z3d what sort of profiling are you thinking?
weko dr|z3d: alive/dead RI
weko We shouldnt use floodfill before we verify what RI is alive
weko I think it is problem in this case
dr|z3d I don't think the attack's over. Seeing floodfill count increase here.
weko dr|z3d: in 5 times for several hours?
weko And go back in 10 minutes?
weko It is attack, sure.
dr|z3d haven't been watching, but I'll add some graphs to keep an eye on the count.
weko See stats.i2p
weko I seen 7-8k floodfill count, while normal values is 1700-1900. And it is while TCSR is 5%. For now it is 25%
weko I writed in attack time, but I forgot about _ in nickname
weko wrote*
weko We need really good profiler and tools for detect such attackers
weko Torrents been not working, some sites been not working.
weko It is big issue I think
dr|z3d sure, it's definitely having an impact.
weko [12:20:30] <dr|z3d> I don't think the attack's over. Seeing floodfill count increase here.
weko Oh, I miss understood.
weko As we can see, different users have different timings in attack ending
weko For now, have you big now of ff?
weko num*
dr|z3d not huge, more than normal. ~2.5K or more, depending on router.
weko Hm, for me and other in #dev (i2pd), attack ended. But not on the one time
weko I see big increase of TCSR after ending for me
weko From 5% to 26%
weko I think reason is algorithms
weko Attack again
weko 12k FFs
weko 13... WTF
weko 15k
weko Tunnel creation success rate: 5%
weko 14k, go back to normal value
orignal I see 6.5K floodfiils
dr|z3d If there was any doubt before that the network was under attack, there should be none now.
weko Sure
zzz I caught one culprit but I think there's more, gotta tweak my logging and restart
orignal we need to unserstand if all these new floodfiils are valid
orignal e.g. their IPs are reachable, they flood and respond to lookups
zzz they are not responding
orignal then it's much easier
dr|z3d yup, that's the problem. just sitting there fielding requests and dropping them.
dr|z3d hence super low build success.
orignal zzz they are not respnding to connect or requests?
dr|z3d lookups, orignal
dr|z3d auto-session ban for ff's that fail to respond to x queries in y interval?
dr|z3d or perhaps even better, carry over to next session as per sybils.
orignal I'm wodering if their IPs are real
orignal and if they are ipv4 I'm wodering where did they get them from
dr|z3d could be spoofed addresses, sure, orignal
dr|z3d a lot of what appeared to be crud ffs weren't resolving via rdns.
orignal no I mean can you establish NTCP2 session with them?
zzz gaah NPE in my logging, gotta restart again
zzz of course I'll work on mitigations but so far java routers seem to be hanging tough
dr|z3d for some definition of tough. part traffic on usually fast routers taken a dive.
dr|z3d also seen firewalled router fail to build any tunnels for at least 1/2 hour.
orignal I don't remeber if I check IP during handshake
orignal but will add
zzz both my routers and the network stats look pretty good
dr|z3d attack wasn't network-wide, seem to hit some routers and not others.
orignal zzz but number of floodfiils
orignal what do you see?
zzz same as what the ilita guys are seeing
zzz <zzz> NTCP2-only, random IP, cost: 3, XfR, random port
orignal I'm asking about number of floodfills on your routers
dr|z3d quantiy?
dr|z3d *quantity?
orignal how many
orignal we see from 5K to 15K now
orignal everywhere
zzz same
orignal so you do see bucnh of floodfill?
zzz yes
zzz same as what the ilita guys are seeing
orignal so are you able to connect to them with NTCP2?
orignal e.g. they publish real IP rather than spoofed
zzz I don't think they are real, but I don't know for sure
orignal then you should see bunch of connect errors in your log
zzz I'm working on mitigations while I wait for my logging to kick in
weko orignal: so, how can I grep my logs for connection errors? Info level
zzz like I said I caught one culprit, looking for more
dr|z3d ~4K floodfills on a firewalled router.
dr|z3d 10m uptime, 0 services up.
dr|z3d 1% build success.
weko 1800, 6 hours, 18%
dr|z3d Floodfill Sort
dr|z3d • Bad: [ZGwjo3] [ZG0tJp] [ZEnMpb] [ZF6Zh0] [ZC~LN5] [ZCxMt3] [ZA3kYm] [ZAqRsL] [ZBBGw7] [ZBF-XD] [ZO~Dp9] [ZOJQf6]
dr|z3d Floodfill Sort
dr|z3d • Bad: [CQvRk8] [CQhEJI] [CQ0zzO] [CQecCp] [CQSTN9] [CRv~-w] [CRkRki] [CR4Xwi] [CROpLE] [CS~NYo] [CWsuky] [CWy1vl]
dr|z3d Floodfill Sort
dr|z3d • Bad: [94UZxs] [95vw0s] [986cQQ] [99Fqt4] [9zDjws] [91ZCwW] [92Y1GV] [93btJX] [9oxPTN] [9oADSS] [9p027k] [9prluo]
weko What with first plot
zzz good news is, we all know what we're doing this week ))
weko Haha
weko Vort checked, and found address
weko 0.176.100.180 0.240.43.157 0.98.68.8 127.11.253.126
dr|z3d all bogons by the looks of things.
weko zzz: are you checking, what when you make connection in ssu2/ntcp2, peer sending you same IP in RI what it actually have
weko I think we shoud
zzz FYI I temporarily shut down stats.i2p
weko zzz: sorry, I meant "receive connection"
zzz if peer is firewalled there's no IP in RI
zzz please give all your ideas to orignal, I don't need any more ideas for Java I2P right now :)
zzz I've coded about 5 mitigations and working on a few more
weko zzz: it isn't my idea
weko It is orignal's idea
weko zzz: if peer is FW, then we don't connect to peer often
weko Are you check or no?
zzz dr|z3d, you have any mitigation code or ideas to contribute?
dr|z3d increasing isj concurrency, perhaps, might help. could check the build fail rate and increase when super low.
zzz I'm thinking the opposite.
dr|z3d could also count consecutive lookup failures from floodfills, and blocklist if they repeatedly fail to respond.
zzz looking up bad ffs just gets you more bad ffs in the DBSRM
dr|z3d we've already got hourly/daily success/failure rates for ffs.
zzz but let me know if you come up with anything
dr|z3d could do something with those.
dr|z3d my preference would be for a banlist for repeat non-responders.
dr|z3d is it worth firing of a periodic ff test on the job queue to validate ffs?
dr|z3d also, what about introducing a minimum threshold for usage? if the router's not been up/known long enough, just avoid it.
zzz I have about 7 mitigations coded and in various stages of testing
dr|z3d ok, when are they likely to land?
zzz "usage" means what here?
dr|z3d usage meaning sending requests for RIs or LSs.
dr|z3d if someone can spin up 4K ff's and the network just blindly starts using them, then we need some speed bumps.
zzz we have some of that now in FloodfillPeerSelector and that's one of the places I'm tweaking
zzz re: land, few days maybe? I want to see where this is headed and also find out if people's routers are OOMing or otherwise dying
zzz also waiting to see if any of my traps spring
dr|z3d any value in extending the sybil scan to look at ffs?
zzz that's all the scan looks at by default
dr|z3d yeah, but it's just checking for ip ranges right now, no?
zzz right, it's doing sybil stuff
dr|z3d and various other attributes, but not success rate of ff replies?
dr|z3d "hostile router scanner"
zzz it's not running often enough to be helpful here imho
dr|z3d anything suspicious about ffs only on ntcp2, no ssu?
zzz makes it more obvious they're bogus, sure
dr|z3d seeing a fair few of those here.
obscuratus Lazy coding, it's easier to spit out fake routers when you only code for one fake transport.
dr|z3d well, if they _are_ bogus, then what about a simple precondition that ff's need ssu and ntcp enabled?
dr|z3d easy to avoid them entirely.
obscuratus Simpler yet, prefer ff's you have actually connected to previously (but we probably already do that to some extent).
obscuratus I think there are legitimate use cases for NTCP2 only. Some providers block UDP (fewer now than in the past).
dr|z3d here are a few ntcp only ffs that all conveniently appear consecutively in the netdb I'm looking at:
dr|z3d tesCZsJ-v1ZNXx7xPTpTWTp1bKyxVgVVf3ix5PTN5~E=
dr|z3d jGp2Kcv1MpOJmUyfdpP7dl~tsFpjsUKDYPJFE0Rjlu4=
dr|z3d GDYXNAXE6HZbH80HSVj3ggiUgOeroB14isFVf0vtvxQ=
dr|z3d kkc2m02ozB6FpgnPMPjFHSrSBZbGcV5JKBH8olbAbPw=
dr|z3d 8k~HA71ZI09WUbWejlw9Sa-4O8D8eUIJ-o0gZdq9t4A=
dr|z3d that's enough to be going on with.
zzz don't need a list
dr|z3d there may well be, obscuratus, but if you're being blocked by your provider, then maybe you shouldn't be running as a ff.
dr|z3d so preconditions could be checked before consulting a ff, and when enabling in the router.
zzz just search XfR
obscuratus If all this attacker has to do is expand there code to fake SSU also, we don't acheive that much.
dr|z3d sure we do, we make it more expensive to run the attack.
dr|z3d and maybe root out some poorly performing bona fide ffs at the same time.
obscuratus On the bright side, we held up pretty well. But it does seem like it's been clearly demonstrated to us that spitting out thousands of fake routers (ff or not) is something we have to consider.
zzz agreed
dr|z3d for (RouterAddress ra : info.getAddresses()) {
dr|z3d if (!ra.getTransportStyle().equals("SSU"))
dr|z3d noSSU = true;
dr|z3d that's about all that's needed to make sure the ff has SSU, no?
zzz that has at least two bugs in 4 lines of code
dr|z3d 1 of them being not checking for ssu2, presumably.
zzz yes
dr|z3d the boolean is defined above, but you knew that so I doubt that's bug #2.
zzz noSSU = true;
zzz for ...
zzz if (SSU or SSU2) {
zzz noSSU = false;
zzz break;
dr|z3d I have to do that if I've already specified boolean noSSU = false; above?
dr|z3d I thought I didn't, and we assume the ff has ssu2 until we check.
zzz as soon as it hits NTCP2 you're setting noSSU to true
zzz but I don't think this is a very useful line of defense, as obscuratus points out that's just the symptom du jour
dr|z3d sure, might not be effective long term, but used with the banlist it'll get a good collection of potentially hostile routers to look at.
orignal zzz let me explain
orignal when you receive SessionConfirmed and find an addesss for "s" with valid IP do you compare it with endpoint?
orignal why NTCP2 only it's obvious
zzz no we don't
dr|z3d 69 floodfills caught within the first 5m of uptime.
orignal I remeber you mentioned that you did
orignal if you don't, may I ask you what's the reason?
orignal when you receive SessionConfirmed you know enpoint
orignal and if you peers published an address he is not connected from it means spoofing
zzz I don't think it's ever been proposed
orignal I'm going to do it
orignal any objections or drawbacks?
zzz it might hurt people using proxies or that just switched their IP... maybe?
zzz also, I don't think it's going to do anything to fix today's netdb spam
orignal people using proxies and publishing thier IPs?
orignal it would
zzz pretty sure they aren't real routers
orignal this idiot wouldn't be able to connect to floodfills to publish himself
orignal do you want me explain how there were created?
zzz and the one router I think I caught was not publishing an IP
zzz sure, go ahed
orignal floodfill not publishing IP? how come?
orignal I have the parameter host= in config
zzz tell me your theory, then I'll tell you mine
orignal the reason of this parameter when you have to use NTCP2-only for some reason
orignal but know your external IP for some reason
orignal honestly this parameter was there from days when I didn't have SSU
orignal but need to oublish NTCP address
orignal so some idiot from kislitsa has generated bunch of floodfilld with random IP in this param
orignal that's all
orignal example
orignal NTCP2\00vhost=127.11.253.126;
orignal caps=XfR;
orignal and only NTCP2 address in FF
zzz ok, so you think it is a real router?
orignal I think it's thousands of real routers
orignal coming from the same real endpoint
orignal and my idea will work perfectly in this scenario
obscuratus Hhhmm, sounds like something simple I could replicate on my testing network.
orignal such moron will not be able to publish himself unless he ....
orignal let's not simplify his life
weko orignal: it cannot be real routers. 127.0/8 not real router
orignal obscuratus 100%
orignal just set any random IP to host in your confim and turn off SSU2
orignal weko routers are real they just publish spoofed IP
orignal how? I explained above
zzz so what, he's running a thousand routers on one computer, say for 10 minutes, then stops them all, changes IP, restarts?
weko orignal: but it can be one routers who just generate RIs with rundom IPs
weko Random*
orignal maybe on 10 computers
weko Maybe
weko Not 1000
orignal zzz he didn't change IP
dr|z3d why not validate the ntcp address published in the RI before attempting to talk to the router?
weko This IPs actually don't run I2P routers
orignal we had them in configs
zzz I mean IP in the RI, not real IP
orignal before start
orignal I think he created many config files with different IP in it
dr|z3d just send a packet to the port, response ? continue : ban.
orignal and start all toghters
orignal dr|z3d do we have such packet peer must repond to?
dr|z3d orignal: must be something we can send that will elicit a reponse.
dr|z3d (or not)
orignal we don't have such message afaik
zzz ok, interesting theory, my theory was that they were not real routers
zzz so let's try to prove one of the theories
orignal tell me yours
zzz not real routers, just an attacker router spamming out netdb stores with fake RIs
dr|z3d orignal: even just testing that the published port is open would be sufficient, without an explicit response.
orignal zzz how does he generates fake RIs?
zzz it would take some custom code. Your theory doesn't need any coding, so maybe it's more likely
weko orignal: choose random IP and generate like any RI
orignal so in your threory attacker is someone who understands i2p internals
weko orignal: it is also possible
orignal and then tries so dumb attack
zzz yes. We know somebody is changing i2pd code for 20 introducers. But this would be a lot harder
zzz I'll try the IP check and see if I catch anything
orignal publishing fake RI is more compliated change than just one const in the code
orignal so do you agree we should check IP?
orignal also one clown at Ilita meantioned that he did
orignal although I don't believe him because he is a clown
weko [23:08:39] <9d54b3orignal> also one clown at Ilita meantioned that he did
weko I think we should not believe him
orignal <Boris> Ну что петушня, как я вас сегодня пошатал, понравилось?))))))))))))))))
orignal <Boris> всего в 2к баксов обошлось) люди заинтересовалсь
orignal weko the fact is that the attack if over
weko For now it is
orignal <Boris> orignal: накоплю денег, будет еще
orignal so we can't exclude that it was really him
weko We had 2 waves, no reasons for will not be 3td
weko orignal: maybe we need verify
weko Ask something about attack
orignal also there another thoery that Boris is Plaz
dr|z3d still some suspect ff's on the network.
orignal weko few years ago Plaz put this irc down
orignal by connecting from 60 destnations
weko Boris said what he spend $2,000 for attack, and what he planned more when he get more money
orignal and it was ElGamal
obscuratus On my testing network, I more-or-less confirmed orignal's idea. Except the i2pd router never published as reachable. So it was just 'L' instead of 'LR'
weko orignal: oh okay, it is maybe plaz
orignal obscuratus nope
obscuratus But it happily accepted a bogus IP address
orignal set nat=false
orignal and host to any IP
obscuratus orignal: kk
orignal and you will see
orignal I know I did this in 2013 ))
weko orignal: in any case we need try to verify Boris
orignal what for?
weko What he is attacker
orignal we need to prevent future attack like this
weko orignal: sure
orignal remeber whole kislitsa are "attackers" like this
orignal and is Boris = Plaz nobody would surprise
orignal everybody knowns Plaz here ))
weko Plaz Plaz Plaz
zzz do I agree we should check IP? It's worth testing, so I'll do that and see how it looks
orignal if you see any drawbacks let me know
orignal we need kinda brainstorming
weko Channels topic say about 7:30 PM tomorrow)
weko Channel*
orignal yes, meeting
orignal but I have some time today
dr|z3d checking the port is open is probably sufficient to kill this type of attack dead.
orignal they might ban you because they see connection with you already
dr|z3d anyways, if my clownometer is anything to go by, this attack's far from done.
orignal that's why I want to add endpoint check
weko And we need add limit of RIs per IP per (for example) hour
orignal how many?
weko "How many RIs can send IP per hour"
weko orignal: 3 for example
orignal how many RIs can be on the same IP?
orignal 3 is too small
orignal people might run many routers in LAN
weko orignal: okay. One router for transit maximum, and don't calculate nontansint and nonfloofll RIs
orignal so with published IP you mean?
dr|z3d LAN, or shared IP on a VPN..
weko No limit for nontransit and nonfloodfills. And limit to one transit/floodfill router
weko orignal: yes
dr|z3d or uni dorm.. plenty of scenarios where 1 IP can host a ton of routers.
weko I think so
orignal especiialy on mobile networks
orignal and Turkmenia has one IP for whole country
weko dr|z3d: 1 ip can host 9999 routers, but for protect tunnels we need limiting count of transiting tunnels to one per IP
weko orignal: it is NAT
weko We don't host many our tunnels on NAT routers
orignal a lot
orignal remeber first hop is connected peer
dr|z3d ~1200 banned ffs and climbing..
weko [23:25:34] <orignal> and Turkmenia has one IP for whole country
weko If it is, I think you cant verify one person/organization control routers in Turkmenia or it is people
dr|z3d orignal's being facetious, weko :)
weko And it is reason why you don't want to host your tunnels with Turkmenia IP
weko It is just make risks for your anonymity
weko It you can host unlimited routers on one IP, then I can just run 10k routers and start analysis
weko If*
weko Floodfills: 3060
weko Again
orignal yes I see too
weko orignal: we can test changes if you have time
orignal will commit SSU2 now
orignal NTCP2 requires more work because yggdrasil
weko Okay
orignal I'm too old and too slow now )))
weko It is sad
weko TCSR decreasing
weko Same scenario, same behavior.
obscuratus I notice many of these bogus XfR routers have 'Compressable: true' in their router info. Another clue that it might be I2PD?
dr|z3d more than likely. that's the russian spelling of compressible.
obscuratus Lol, I just fat-fingered it as I transposed it here.
dr|z3d them some fat fingers.
dr|z3d just watching the clownometer keep climbing in the sidebar (aka the banned peer count)..
dr|z3d build success currently @ 50%
dr|z3d ~2.7K banned peers now. no signs of slowing down.
dr|z3d you see, my theory is that this may be some not very elaborate attack that dies as soon as ntcp-only ffs are blocked.
dr|z3d so I don't see there being much downside to just banning them. low cost for us, higher cost to circumvent.
dr|z3d I've also increased the threshold for trusting ff's from 3->8 hours.
dr|z3d and just for good measure, L and M tier ff's no longer pass muster and are deemed slow.
dr|z3d which reminds me, unreachable floodfills should also be discarded.
orignal where do you see "Compressable: true"?
zzz got the check running, caught two so far, both java floodfills
obscuratus orignal: When I do a NetworkDB List (I'm on a java-based router FYI).
obscuratus And the correct spelling should be: Compressible: true
orignal so it's zzz's misspelling
orignal and i2pd always creates compressible identity now
obscuratus I think I'm the one who did the misspelling. :)
zzz yeah I'm in the clear. obscuratus that's just an advanced console thing, it's not actually in the RI
dr|z3d > my profuse apologies, orignal. :)
Xeha dr|z3d: banning ntcp-only routers is fighting the sympton, not the cause. the attack can just publish ntcp and ssu
orignal and ntcp2 only is valid thing
dr|z3d valid, but rare, even more so for floodfills.
dr|z3d it's a temporary fix, anyways, just to see how much crap gets caught.
Xeha no its not rare, i got several FFs across the world which are ntcp only due to nasty state firewalls
Xeha or datacenter firewalls
orignal or UDP is blocked
dr|z3d small price to pay. with anything from 3-15K dodgy ffs out there, I'll take the loss of a few good ones for now.
orignal I had another suggestion for zzz long time ago
Xeha there's a difference if you want to do it just on your routers, or push it as a whole i2p+ update
orignal a floodfill must be on both ipv4 and ipv6
orignal and reachable through both
Xeha most of my FFs arent on ipv6 :(
orignal then don't let them be a FF
dr|z3d well, it's shipping as an update until more robust mitigations come along.
dr|z3d and what orignal said. no actual requirement to be a floodfill.
Xeha yea, i get the idea. you'd want floodfills to be as reachable as possible
orignal yes for not the requirement is reachable thtough ipv4 or ipv6
orignal I would requie both
dr|z3d not disagreeing with you, orignal, I was saying to Xeha he doesn't _need_ to run his routers as floodfills.
orignal ofc you don't need to run a floodfill
orignal run a floodfill only if you can
dr|z3d if your provider is blocking SSU, maybe not the best idea to run one.
orignal it still can work trough NTC2
dr|z3d for the same reason you're suggesting ipv6.
dr|z3d because ffs work perfectly fine over ipv4.
orignal but you don't know the reason, really
orignal because I didn't tell it yet
dr|z3d it's a higher barrier to entry.
orignal no. it's about OBEP and IBGW
orignal I don't like that ipv4 is required for them
orignal and recahble ipv4 for IBGW
dr|z3d ok, not sure I follow, but you obviously have some logic to support your proposed ff criteria.
orignal we must pick ipv4 routers for OBEP and IBGW for compatibility with floodfiils
dr|z3d so you'd remove that requirement by obligating ff's to have both ipv4/6. I think we're probably near the point where that would make sense, but for now ipv6 is still is still experimental as far as uptake is concerned.
zzz i'd tell you what % of netdb was v6 if my netdb wasn't so borked right now
zzz very very few routers caught in the IP check so far after testing for an hour
orignal what do you mean?
orignal you beleive those IP are valid?
zzz no, I just mean my stats are off
zzz but iirc it was only like 30%
orignal I'm talking about Ip check
obscuratus I've thrown together a quick bash script on my testing network to generate a I2PD router with a random IP address similar to original's suggestion above.
obscuratus Right now, I'm running it in a loop every 20 seconds (I'm not sure how quick I can go, and still contact the other FF on my testing network).
orignal because no IP check yet
obscuratus I can only get 'Xf' and not 'XfR', but if needed, I can probably make a patched I2PD router.
orignal I'm doing the code right now
zzz I screwed up the ygg check, trying again
orignal Xf vs XfR
orignal I can tell you why
obscuratus So, I'll be able to test mitigations on my testing network if anybody wants.
orignal because you are running trunk
orignal that guys probably run release
orignal and yes good finding about R
zzz also seen case where the low-end of the IPv6 is different (temporary addresses)
obscuratus orignal: Yeah, I build from git.
zzz so might need to only check first 8 of ipv6
orignal I have chaged that logic recdently
orignal zzz, please explain why it should be different?
orignal you report their address in SessionCreated
orignal current one
eyedeekay Is major down?
zzz orignal, ipv6 temporary addresses
orignal maybe
obscuratus eyedeekay: Lot's of bogus FF's right now, so you may just be having trouble getting to it.
orignal yes but why it should be different?
orignal between SessionCreated and SessionConfirmed
orignal even if it's tempopartu
eyedeekay Yeah I'm seeing that, trying to get caught up on what's going on
orignal *temporary
zzz it takes a while to 'find out' your ipv6 changed
orignal if new address comed in SessionCreated
zzz yeah but we don't rebuild a new RI between session created and session confirmed
zzz example:
zzz net.i2p.data.DataFormatException: IP mismatch actual IP 2001:470:28:365:216:3eff:fedd:2fa1 in RI: [RouterInfo:
zzz [host] = [2001:470:28:365:0:0:0:38]
orignal so how many bytes should we compare?
orignal thanks
zzz so far none of the traps I've set have really yielded much