obscuratus
orignal: I think that's what we're intending to do already. If the Inbound Message Distributor receives a DSM with forwarding instructions (that's delivery type Router, right?) coming down a tunnel, we going to drop that message.
obscuratus
Or, at least, I've proposed that. It hasn't been merged, but I think it's the same thing as you're talking about.
obscuratus
orignal: I had thought I2PD already did that, no?
orignal
ofc we do for IB
orignal
but I'm talking aboue OBEP
orignal
if somebidy send it through an aoutbound tunnel
obscuratus
Heh! I guess I'm still learning. I thought the OBEP would only talk to IBGW. I didn't realize it would do that. :)
orignal
how do you publish you RI to an FF if you can't reach it directly?
orignal
the only way is through tunnel
dr|z3d
iirc, java does non-encrypted datastores for slower (arm/android) routers, but I may be wrong.
dr|z3d
ofc, we could ensure they encrypt.
obscuratus
orignal: Yeah, I think we send it out an exploratory tunnel.
obscuratus
So, this proposed change would then eliminate the ability to send your RI down an exploratory tunnel, right?
obscuratus
Or, does it arrive at the OBEP as a garlic message?
obscuratus
orignal: You've got me curious. I'll add some logging.
orignal
gents you ban real routers
eyedeekay
Yeah it's obviously happening, we need to get the fake ones out of the netDB before the sybil analysis sees them, working on it
eyedeekay
We can at least cut this down significantly I think
orignal
as result tunnel creation rate is low when even no attack
eyedeekay
Yeah it's resulting in observably fewer floodfills even after an attack because we have to wait for bans to expire, we could think about expiring them faster but I'd rather try to stop the false positives
eyedeekay
Oh great, looked at my graph, just saw it climb from 1200 to 1800 then the bans started
eyedeekay
Let's see where this goes over the next 20-30 minutes
orignal
no, ther are attacking duiring our nighttime only
eyedeekay
Hm. Wonder why I would be seeing it change like that then
eyedeekay
Never mind, stabilized around 1650 floodfills known to me, real or not
orignal
they found out that you, me and dr|z3d are in EST
mesh
orignal: yeah they attack during a specific time winow
mesh
I sort of wonder why
mesh
what's in the latest i2p+ update that went out today? (why is there no changelog... blog entry... argh)
orignal
two theories
orignal
then want to attack when we are offline
orignal
or it just thier working hours
mesh
(when you're offline)
orignal
during my night of course
eyedeekay
Those are not mutually exclusive
mesh
but yeah. the attack can be sorta mitigated simply by restarting the router
mesh
I've also found the attack can be mitigated by creating a very agggressive flood fill
orignal
there keep changing strategy every day
orignal
*they
mesh
orignal: for the past two days they've gone with the forged ri attack
orignal
eyedeekay what do you think about forbidding database store message on OBEP?
orignal
they forged them differently
mesh
it's quite effective and is likely quite cost effective because it tricks good routers into doing all the work
orignal
initially they just point whole rouyter content
orignal
now they chnage s key
RN
the sybil test is part of mesh's strategy, eyedeekay what is going wrong when the sybil test gets false routers?
orignal
this night they clone only particular floodfills
mesh
orignal: why does that matter...
obscuratus
RN: I tricks Sybil into banning routers and whole IP addresses.
mesh
the attack can still be mitigated by creating a very aggressive floodfill router
obscuratus
How do you make your floodfill aggressive? Testosterone shots?
mesh
I upped the exploratory tunnel count to 32 (even though the webconsole maxes out at 16 tunnels), I decreased peerTestTimeout to 250, I run sybil every hour but actually raised the minimum threshold for block to 50, and of course force all the routers into being floodfills
eyedeekay
Yeah that's not a thing I'm familiar with either, can you refine your terminology? Like are you saying "manually enable floodfill" or "offer lots of bandwidth" or "use restrictive sybil-analysis settings" or something like that?
eyedeekay
Oh never mind you explained
eyedeekay
Thank you
mesh
the result is a router that is very well integrated into the network. The share ratio is cut in half but it resists the forged RI attack. During today's attack my bsr never dropped below 40%. All my i2p web services continued to be available
orignal
asnother thing we have noticed
orignal
they copy family signature with all field
mesh
it sort of makes sense: if your router is very aggressive in exploring the network and build up its own authoritative view of the network, it's resistant to being flooded with fake news about the network
orignal
and of course it fails for fake routers
orignal
just FYI
eyedeekay
IMO a failed family sig should be an immediate ban, no reason for it that is non-malicious
orignal
yes, what I did now
mesh
eyedeekay: what about the stormycloud routers?
orignal
so for strimycloud your alsways know which one is real
eyedeekay
Fair enough, there is a model defense laid out on the netDB page where everybody manually enables floodfill, as "fight sybil with more sybil" yours is an interesting take on that
orignal
one with good signature
eyedeekay
Bad sig is a bad sig, if it's a mistake on StormyCloud's part, they should fix it but in all likelihood it's an attack
orignal
they have fixed long tim eago
orignal
if I remeber
eyedeekay
Good
orignal
but clones of strormycloud routers contain original one
orignal
that's obvously fails for another router ident
mesh
eyedeekay: yeah exactly. aggressive floodfills are very difficult to be knocked offline. During the attack I saw virtually no degradation in the availability of my I2P services. In fact they seem to be better performing because I reduced peerTestTimeout to 250. The tradeoff is that the routers with this configuration (1) consume a lot more cpu resources and (2) they share a lot less bandwidth because most of
mesh
the bandwidth is consumed building exploratory tunnels
orignal
ofc in few days they will discover it
mesh
eyedeekay: you know the attack happens so suddenly that I wonder if the router couldn't detect it. Today I saw floodfill count jump by thousands in less than an hour
mesh
one simple heuristic would be to track the number of floodfills that are all showing up wwithin a 60 minute window and punish them
mesh
what I always notice (I'm usually awake during the attacks which happen in my mornings) is very quick rise in floodfill count followed by a very quick rise in the banned count. Banned count can climb very high -- 20,000+. Usually it sits below 2,000. From a heuristic perspective it's very clear that things are not normal and the network is under attack
mesh
orignal: btw are RIs signed by the floodfill that publishes them?
orignal
no
orignal
only by themselves
mesh
orignal: so there's no way to tell the provenance of an RI? like which floodfill it came from?
orignal
no
mesh
hmm yeah that's a problem
mesh
even if we can heuristically identify the suspicious floodfills we can't ignore their forged RIs can we
obscuratus
orignal: Regarding dropping RI DSM at OBEP, we would want to check and make sure we actually have code that does this, but didn't realize it.
obscuratus
If this is something we never do (canon and cannon), and something you never do, then when we see it, it seems like it could only be malicious.
orignal
no you shouldn't
dr|z3d
in java, we can validate a router's ip address with geoip.
mesh
dr|z3d: it's not exactly 100% accurate
obscuratus
orignal: That's why I modified my logging. I wanted to check on my testing network, and see if I ever get those.
mesh
dr|z3d: but it could definitely become another factor
obscuratus
How would we use GeoIP?
dr|z3d
at this stage, best effort trumps 100% accuracy.
dr|z3d
we check if the published ip address for the router validates via geoip, obscuratus. if it doesn't, ban the router.
obscuratus
Oh, so you just checking if the IP is somewhere in the range of the GeoIP list?
dr|z3d
that ends up pre-empting sybil bans, so we retain most of the valid floodfills.
mesh
dr|z3d: you might restrict such a check to floodfills. it would be another constraint on floodfills.
dr|z3d
you query geoip directly, obscuratus. getCountry() or thereabouts.
mesh
dr|z3d: I don't rememmber if the fake floodfills had valid geoips but it seems reasonable
orignal
but now it's useless becuase fake routers has real IPs
mesh
dr|z3d: also as I mentioned above, all these fake floofills come online at the same time. I would recommend punishing floodfills that are themselves part of a flood. perhaps more than X other floodfills all joined within 30-60 minutes
dr|z3d
not useless, orignal. quite effective.
RN
(side track: if major is a logging bot, why does it have voice?)
mesh
(side track: if major is a logging bot we should burn it at the stake)
RN
lol, be nice mesh.
dr|z3d
we're looking up the transport ip, orignal, not the RI published ip.
dr|z3d
and therein lies the weakness of the current attack.
dr|z3d
see FloodfillPeerSelector, mesh, for an insight into how floodfills are grade.
dr|z3d
*graded
dr|z3d
obscuratus: public String getCountry(Hash peer) in CommSystemFacadeImpl
dr|z3d
on a floodfill here, just shy of 1200 active floodfills, 7.5K banned routers.
dr|z3d
build success at 70%
mesh
I've got similar numbers. 1200 floofills with 8.6k banned routers
dr|z3d
and sybils meeting the ban threshold? 0.
mesh
dr|z3d: is there a place in the code where a new floodfill is detected for the first time?
dr|z3d
as soon as a new RI is acquired.
dr|z3d
as for blogging, mesh. no. sorry. don't have time for that. read the commit logs if you want insight into what's changed. don't be lazy, or entitled.
dr|z3d
RN: more a status thing with major, but the occasional random intervention is amusing :)
dr|z3d
as for targeting floodfills, current attacks aren't solely utilizing floodfills.
mesh
yeah I think part of the problem is that even if you know all the bad floodfills you can't identify RIs
mesh
that came from those bad floodfills
dr|z3d
once you've banned the floodfill, you won't receive bogus RIs from said floodfill.
obscuratus
dr|z3d: Until we address issue 397, we can't be sure that floodfill isn't forwarding that RI for someone else.
dr|z3d
just set your global logging to warn, mesh, and enjoy the show.
dr|z3d
that's a separate issue, obscuratus. all I'm saying is that banned floodfills can't forward RIs for anyone.
obscuratus
They may be a perfectly legitimate FF, but we just forwarding the RI for someone else.
dr|z3d
sure, good floodfills will happily send shit RIs all over.
obscuratus
Yeah, until we fix 397
dr|z3d
so really we need to address the issue at both ends.
dr|z3d
we check RIs on receipt, and floodfills check RIs before send.
dr|z3d
there's some laxity wrt to what floodfills will send, RI-wise. I've tightened that a little.
obscuratus
I'm leaning towards orignal's solution. Have something like a white-list for known-real FF. The others will wait their turn until we can contact them.
dr|z3d
in canon, routers that are considered perm-banned will be distributed by floodfills.
dr|z3d
*aren't considered
dr|z3d
sure, a whitelist could work, but we already have something along those lines already with the ff selector.
mesh
what is 397? should floodfills republish and re-sign RIs? Seems like it would be good to know the true floodfill 'origin' of an RI
obscuratus
Yeah, but so what. As long as the receiving router has a list of known-real FF, it will also wait until it has contacts that FF (or not).,
dr|z3d
we already profile floodfills, so using that as the basis for whitelisting is what I'm driving at.
dr|z3d
I'd be just as happy seeing a session-persistent blacklist for bad floodfills in similar vein to the sybil detection.
obscuratus
If querying the peer profiles is fast enough, that might be a solution. But we might need a quicker in-memory method purpose built.
dr|z3d
profiles for active peers are help in memory.
dr|z3d
*held
obscuratus
I'm coming around to something similar to orignal's view on banning. It really looks to me like some one or several people are actively expoiting flaws in our banning heuristics.
dr|z3d
they're trying.
dr|z3d
not getting very afar here, for reasons I mentioned above.
dr|z3d
*far
obscuratus
If they're causing good floodfills to be banned, they're succeeding.
mesh
obscuratus: that is the nature of the attack
dr|z3d
orignal's estimate is around 1.2K good floodfills, which is consistent with what I'm seeing on various routers.
mesh
the attackers generate a lot of fake floodfills. they publish garbage. this is followed by dramatic rises in the banned count
mesh
it's a cheap and effective attack I bet, more so than the previous attempts to ddos the network by sending lots of data
dr|z3d
as I mentioned, obscuratus, sybil scans aren't banning anything here.
obscuratus
I'm scrolling back, but can't find it. If I said sybil was the only banning problem, I mis-spoke.
mesh
I see lots of "Floodfill with SSU disabled" "LU and older than current version"
obscuratus
As it is, running once a day, I'm not at all sure it's effective for it's desired purpose.
mesh
in the ban list
mesh
very few say "Sybil Analysis" ... I assume those routers were banned by the sybil
obscuratus
Do we even have a reason listed for our IP bans?
obscuratus
The IP bans are probably our achilles heel with respect to banning good FFs.
dr|z3d
blocklist, no, session bans, yes.
dr|z3d
1d? I can't remember if I tweaked I2P+ defaults, but locally it's @ 3h.
mesh
mayhaps there should be an "unban all" button btw
mesh
you can unban inividual peers but of course that's silly when you've got 20,000+ banned routers and many of them are actually good routers
dr|z3d
if you've got 20K bans MOST of them are shit routers.
obscuratus
I look at it from the perspective of controller theory. When you're trying to control a parameter with discrete measurements, the polling period needs to be much faster than the response time. Ideally, instantaneous polling.
mesh
dr|z3d: I think there's lots of good routers mixed in with the bad routers
mesh
for example, the "Floodfill with SSU disabled" probably are bad routers
obscuratus
Which reminds me, we also need to address the issue we ran across a few days ago, where someone was spoofing RI that which we couldn't discern from real RI, and we're banning that router.
obscuratus
So, as it stands now, easy, just spoof a FF router ID, but without the SSU transport, and we ban that router without realizing there's a good RI out there for that router.
dr|z3d
I can only suggest you take a look at how I2P+ handles the problem, obscuratus, and then take a view on its efficacy.
obscuratus
But, I feel confident we will fix that one.
dr|z3d
good floodfills without SSU are few and far between.
obscuratus
What if it really has a published SSU transport, but we're looking that the spoofed RI that wrongly says it doesn't have a SSU transport?
mesh
I wish there was a way to record the attack and replay it. Now I'm genuinely curious to know which comes first.
RN
LOL major
dr|z3d
check for a valid ip before all else.
mesh
dr|z3d: how to check for a valid IP?
dr|z3d
only when the ip is proven valid (geoip-resolvable) do you then eliminate ssu-disabled floodfills.
obscuratus
The spoofed RI's had the same IP as the real RI.
dr|z3d
transport ip, NOT RI ip.
dr|z3d
I pointed you at the code in question, obscuratus.
obscuratus
The RI is how we know their transports.
dr|z3d
so I shouldn't need to be explaining this :)
dr|z3d
not the published ip, the ip being actively used I think is how we're doing the lookup.
dr|z3d
/**
dr|z3d
* Uses the transport IP first because that lookup is fast,
dr|z3d
* then the IP from the netDb.
mesh
obscuratus: this is where the whitelist would kick in?
obscuratus
mesh: We're really talking about two separate problems actually. We'll probably fix the issue with spoofed IPs.
obscuratus
The second issue may be able to be quickly resolved by tweaking our method of building a list of FFs.
obscuratus
If the check of 'last heard from', or something like that, in peer profiles is quick, just add that screener to our ff selector.
mesh
obscuratus: It sounds to me like the fundamental problem is: "I'm looking at a RI for a suspicious Floodfill. But before I ban this Floodfill how can I verify that the RI I'm looking at is not a forgery?"
mesh
at least when you put it like that then the whitelist makes sense. that's just a reputation attack. but there are many floodfills who we can be pretty sure are not bad actors even if you see a suspicious RI that somebody published about them
dr|z3d
it's fast, but we don't profile floodfills on sight, we profile them on a schedule.
dr|z3d
or rather, we grade floodfills based on their profile data on a repeat schedule.
dr|z3d
if a floodfill is too fresh, poorly responding, we put them at the bottom of the grade table.
mesh
there are floodfills out there that are my homies. there's one estonia that stuck with me through thick and thin. I would never want to ban him just because I see a suspcious RI with his IP