#ls2 (irc2p) | #ls2 I2P Protocol Development Discussion - Next Meeting Monday, October 9th 6:30 UTC

zzz good morning orignal

orignal back

zzz I wanted to fill you in on some advice I've gotten

zzz I've enlisted the help of an outside company to assist us

zzz So far, what I've received is pretty generic and high-level

orignal go ahead?

zzz but it's still valuable to get outside opinion

zzz and it helps us organize and focus our response

zzz there's 3 categories of suggestions

zzz all 3 we have in Java i2p already, but they need fine-tuning and improvement

zzz but I suspect all 3 need major work on the i2pd side

orignal tell me

zzz here we go:

zzz 3 ways to counteract ddos attacks:

zzz 1) Rate limiting

zzz rate limits help slow down ddos. can be done at several locations and protocols within the router

zzz 2) Traffic filtering

zzz block by IP and router hash

zzz avoid distributing peer info about blocked ips/routers to other routers

zzz 3) Peer discovery and verification

orignal we have it now

orignal say we ban unreachable routers for 2 hours

zzz classify and validate new peers, don't spread unvalidated info around to other routers

zzz EOT

zzz pretty generic, but a good way to structure things

orignal we have 2 and 3

zzz you don't have a ban-forever IP or router hash list though, do you?

orignal remeber for 3 I verify IP now

dr|z3d definitely don't have rate limiting/throttling either, no?

orignal and if doesn't match wih actual one I close connection

orignal I ban for 2 hours

zzz right, you do have some 2) and 3), for sure. But we both need to improve things

orignal not ban just exclude from my netdb

orignal ban by IP maybe

zzz you need a permanent IP and hash ban list

orignal I had it in NTCP many years ago

zzz we both need to do a lot better in 3)

orignal it's easy I can do it

orignal about 1

orignal well I'm not sure if it's a right thing

orignal because you don't know the rate

zzz all things need limits. The hard part is figuring out where to do it and how to measure

zzz there's really two parts for us:

orignal reember you are always limited by your network port

zzz 1a) don't get overloaded yourself. That's usually not a problem for i2pd, it's fast and computers are powerful these days

zzz 1b) don't send overload to others

orignal but you need a difinition of "overload"

zzz right. 1) is the hardest

zzz 1a) "overload" for yourself isn't so hard. Queue sizes or queue overflow or latency are indicators of overload

zzz 1b) not sending too much out is harder. Throttles or rate limits based on typical patterns can help

orignal for myself yes it's easy

zzz also there can be ramp-up detection, where you don't allow the rate of something to increase too quickly

dr|z3d like # of transit tunnels, for example.

zzz an example of 1b) is where we drop tunnel build requests rather than reject or accept under certain situations

zzz so we don't propagate the traffic

zzz that's all I got, if I get more later I will pass it along

zzz I thought it was helpful just to organize our work a little

orignal do they have concrete reccomendations?

orignal about current situation

dr|z3d I want to air an idea, an informal proposal, regarding classes of routers.

zzz not yet, maybe later, may take a while

orignal basically about the source of the attacks

dr|z3d let's say we monitor different classes of routers with the same caps, eg XfR, LU etc..

zzz I'm not sure how much they can do or when. They are busy also

orignal can they evaluate how much power is needed

orignal simply speaking

zzz ideally, yes, they could do that for us. I'm hoping

dr|z3d with a per-classes throttle using buckets, it might make sense to drop/reject requests from a given class in a given timeframe if they ramp too much.

orignal is this attack comes from one moron or from an organztion with bujdget

zzz right orignal I've asked those questions

orignal for me it's most impotant question

zzz we may have to motivate them with $$ to do it

zzz don't know yet

orignal "salt" guys claimed he did it to write his unversity degree paper

orignal but we don't believe him

zzz interesting :)

zzz ask what university so we can file complaint :)

dr|z3d if we only knew the university.

dr|z3d exactly my thoughts, zzz.

dr|z3d so, buckets for classes of router.. any thoughts?

dr|z3d if someone's spinning up a few thousand routers on a whim using a standard profile..

orignal you can file your compait to sportloto.i2p with the same results

orignal have you heard that Russian hackers can't be prosecuted in Russian if they do it under name of Russia?

dr|z3d we should be able to detect and defend against that.

orignal it's just FYI

zzz orignal, do you pick U peers for tunnels?

orignal yes, for middle peer

orignal and for OBEP

dr|z3d U needs to die. U is for useless.

orignal I don't pick it from IBGW

orignal I would say U and R are useless

orignal I don't rely on them

orignal and rely on addresses only

dr|z3d if a router's firewalled, it's best avoided for tunnel building. more hops, less reliability.

orignal I disagree

dr|z3d if a floodfill's firewalled, then it should be banned outright.

orignal I convert it to ordinary router

dr|z3d now is the time to start being a bit brutal with routers that aren't behaving as expected. U class routers, mammoth shit, as you'd say, orignal

orignal U class routers are most ordianry users

orignal and they do need transit for anonymity

dr|z3d downgrading a firewalled floodfill and removing the f cap sounds like a good idea. zzz..

dr|z3d they should be tolerated, but definitely not used for tunnels.

dr|z3d that's their problem, not ours. want transit, sort out your firewall.

orignal be realisitic

orignal and many of the can't

orignal because they use mobile networks

dr|z3d this is realism. the current network attack makes for some hard choices and a lot less latitude.

zzz orignal, dr|z3d made that change a long time ago. I resisted

zzz he is right

zzz I looked at part. tunnel stats for U routers

zzz it's an average of about 1 per hour

zzz I made the change last week

zzz I think it's going to be a big help for build success

zzz 2nd benefit: we don't force the previous hop to lookup the RI for a crappy U router

zzz dr|z3d, what's the benefit of buckets? I don't get it

orignal please explain your difinition of U

zzz U cap

orignal because when I pcik peers I don't look at it

orignal I only see if next peer can be reached from previus

orignal that's all I do

zzz yeah my recommendation is to not use U routers in any tunnels, client or expl.

orignal this is dicrimination of U users

orignal they need transit

zzz I looked at the stats. U routers have almost no tunnels now. Average about 1 tunnel

orignal lol

dr|z3d the definition of a U cap peer is one that's using introducers.

orignal ask _mblw_ how much transit he has ))

orignal they have a lot

zzz here's an example:

zzz NU router:

zzz stat_tunnel.participatingTunnels.60m = 1.43;5.54;82.79%;555;555;555;

zzz that's an average of 1.43 part. tunnels over the last hour

orignal again you can ask around people running U routers

zzz two more NU:

zzz stat_tunnel.participatingTunnels.60m = 9.72;42.35;137.21%;555;555;555;

zzz stat_tunnel.participatingTunnels.60m = 2.47;7.10;107.67%;555;555;555;

zzz LU examples:

zzz stat_tunnel.participatingTunnels.60m = 0.15;0.15;270.15%;555;555;555;

zzz stat_tunnel.participatingTunnels.60m = 0.00;1.04;0.00%;555;555;555;

zzz stat_tunnel.participatingTunnels.60m = 0.08;0.75;23.48%;555;555;555;

zzz stat_tunnel.participatingTunnels.60m = 0.54;0.62;93.23%;555;555;555;

zzz stat_tunnel.participatingTunnels.60m = 2.03;5.04;45.40%;555;555;555;

zzz stat_tunnel.participatingTunnels.60m = 1.42;4.64;72.60%;555;555;555;

zzz most LU are under 1.0

zzz that's why I know dr|z3d was right about this

zzz was trying to be nice and give the U routers some cover traffic, but it's not working

zzz and we can't be nice right now when we have big tunnel build problems

orignal then why it works for i2pd users ?

zzz how many tunnels do you have through U routers right now?

orignal _mblw_ says like 30-40

orignal on his router

orignal *** afk ***

orignal be back in 2 hours

zzz ok

zzz <zzz> dr|z3d, what's the benefit of buckets? I don't get it

dr|z3d you detect abnormal ramping of certain classes of routers and throw them in a bucket.

dr|z3d x new XfRs in 10m, for example.

zzz what do you do with the bucket?

zzz and ramping of what?

dr|z3d fill it with any given class of routers.

dr|z3d XfR, LU, whatever.

dr|z3d when bucket's full, reject connections/tunnel requests from said class.

zzz so its a leaky bucket?

dr|z3d a bit like the connection throttler in the tunnel manager. movable, leaky, call it what you will.

zzz so the theory is an attacker fleet would all be of the same class?

dr|z3d yeah, if it's some sort of scripted attack.

dr|z3d "create 2000 routers from this config"

zzz and how do we set thresholds for each of these buckets?

zzz I have 36 different combinations of caps here

zzz and congestion caps will triple that

dr|z3d forget congestion caps.

dr|z3d we're really only interested in 3. b/w tier, ff, and reachable/unreachable.

zzz ok so I strip congestion caps

zzz ok so I split out each cap separately? so a XfR router goes in the X bucket and the f bucket and the R bucket?

dr|z3d I was thinking about combos.

zzz thats what I'm saying. I have 36 combos

dr|z3d XfR would be one bucket, XfU another.

zzz 664 NfR

zzz 461 XfR

zzz 458 LR

zzz 355 PfR

zzz 304 XR

zzz 274 PR

zzz 127 NR

zzz 86 LU

zzz 83 OR

zzz 65 OfR

zzz 49 PU

zzz 39 XU

zzz 26 MR

zzz 17 XfU

zzz 15 NfU

zzz 12 P

zzz 9 X

zzz 9 PfU

zzz 7 OU

zzz 7 NU

zzz 6 L

zzz 3 MfR

zzz 3 LfR

zzz 3 KR

zzz 2 XOfR

zzz 2 POfR

zzz 2 Nf

zzz 2 N

zzz 1 Xf

zzz 1 POR

zzz 1 Pf

zzz 1 OfU

zzz 1 O

zzz 1 MU

zzz 1 KU

zzz 1 K

dr|z3d PO/XO = X.

dr|z3d buckets could be dynamic, too. until there's x percent of a given combo, don't bother creating a bucket.

dr|z3d K we don't care about, not enough of those around.

dr|z3d and they don't do transit in any event.

dr|z3d if they happen to get to be a problem (unlikely, but possible), we just ban then as a class outright.

dr|z3d routers without a U/R cap can go into one bucket perhaps?

dr|z3d so we're whittling down the number of possible combos.

zzz precise number of buckets doesn't matter. It's several

zzz for each bucket do we have to set the limit manually? or they all have the same limit? or it's automatically adjusted?

dr|z3d ok

dr|z3d automatically adjusted based on the size of known peers in netdb, no?

dr|z3d with perhaps different weights for different combos, slower routers get smaller buckets.

zzz and we'd have these checks at (at least) 3 points? NTCP inbound, SSU inbound, and tunnel build requests?

weko [14:11:57] <zzz> I looked at the stats. U routers have almost no tunnels now. Average about 1 tunnel

weko For my very low uptime firewalled router I transit 1/3-1/2 of all traffic. I think not bad. In tunnel count, about 5-15 on average

dr|z3d you're the low level guy, zzz, whatever you think will a) work best b) not slow down the router and c) not hit the ram too hard.

zzz ok I think I know enough to respond:

zzz overly complex, very difficult to tune/tweak/test, and overly focused on the attack du jour

zzz however I am going to look at our overall inbound conn throttles in ntcp and ssu

zzz I think they need tightening

zzz because I see routers getting a thousand i/b conns in two minutes after startup

zzz I thought we had throttles in both places

zzz we don't need buckets there

zzz also for IB conns you don't have caps until the handshake is complete, that would be the wrong place to throttle

zzz thanks weko

weko Introducers really helpful

dr|z3d ok

dr|z3d if we forget about f and R/U and just focus on bandwidth tiers, still overcomplex, zzz?

orignal I would rather split by combiation of addresses

orignal guys, let's back to LU issue

orignal what's your problem?

orignal low tunnel creation rate or what?

orignal if we exclude LU from transit, then once I see a LU router next it tunnel it means it's owner of IB tunnel

orignal profit !!!

zzz hmm

zzz forgot about that

zzz but I think it's already that way because success rate is so low

orignal I have 50-60% now

orignal do you think it's low?

weko 30%. No low also.

weko Not*

weko We need more investigation before do something. More discuss, anyway. I suggested attack model, why I don't right? You thinking what you right, but please see other side of your changes

zzz maybe for exploratory only

zzz definitely not client

zzz and maybe only 10% of the time or something

zzz we're talking exploratory, they wouldn't be in client tunnels anyway

zzz you need to know your exploratory build success rate, not combined

zzz thats what I'm doing weko

weko We need more hard network changes for general fix

orignal but why do you need to care about exploratory rate?

zzz because that's what will improve a lot if we skip U

dr|z3d 60-80% here :)

orignal then skip it for exploratory only ))

zzz U isn't in client tunnels anyway, most of the time

orignal but remember that the main goal of I2P is anonymit, rather than rate

zzz sure, but it has to work

weko I have some ideas about "many IPs" protection also, about Sybil protection, about improve "good" user experience... Please see us and me!

orignal please exaplain what's wrong with U in client tunnel?

orignal I would say opposite

orignal U is usually on home PC

dr|z3d re buckets, here's an even simpler proposal. 1 bucket for U peers.

orignal e.g. powerful box

weko As I writed in i2pd's FAQ "all TCSR >10% is good"

zzz in practice, even if we allow U in client tunnel, they aren't in fast profile tier

orignal R is usually on VPS witj limited resources

zzz it just doesn't happen very often

orignal think about this idea

zzz there's very few MU/NU/OU/PU/XU. They're all slow LU

weko Maybe not

weko My PC maybe powerful and 24/7 uptime

orignal ofc I exlude LU from client tunnels because L

orignal not because U

weko But I U

weko We should not profile by R/U flags, it just for mention, need we introducers or no

weko Need we ask introducers*

weko Also I think we need general and shared paper/proposal/recommendation about profiling

weko With anonymity index (user can choose)

weko Anonymity index mean how many peers we choose for our tunnels

weko But this index affect quality and speed ofc

orignal zzz, regrardless this problem I see another issue with SSU2

orignal say I have an SSU2 session and thier RouterInfo was good in SessionConfirmed

orignal later this RouterInfo was expired in my netdb, but peer didn't send an updated one yet

orignal but sent a PeerTest

zzz so ask him for it

orignal how?

zzz send him a database lookup message :)

orignal so I can't send Alice's RouterInfo to Charlie because it's not in my netdb anymore

orignal how do you resolve this siatuation?

orignal lookup? what if he is not a floodfill?

zzz he should always respond to a lookup of his own RI

orignal hmm something new

weko Need do many "no backward compatibility" things now, for enable "no backward compatibility" part when we will be ready for this

orignal I thought non-FF shoudl always reply with closests FFs

orignal but how do you resolve it?

zzz this is exception

orignal why?

orignal it's not rare

zzz because it's direct, not through tunnel

zzz here's our "direct lookup" code:

zzz DatabaseLookupMessage dlm = new DatabaseLookupMessage(ctx, true);

zzz dlm.setFrom(ctx.routerHash());

orignal you mean for lookuo

zzz long exp = ctx.clock().now() + 5*1000;

zzz dlm.setMessageExpiration(exp);

zzz dlm.setSearchKey(_key);

zzz dlm.setSearchType(DatabaseLookupMessage.Type.RI);

zzz OutNetMessage m = new OutNetMessage(ctx, dlm, exp,

orignal fine

zzz OutNetMessage.PRIORITY_MY_NETDB_LOOKUP, _oldRI);

zzz ctx.commSystem().processMessage(m);

orignal I'm asking about SSU2

zzz doesn't matter, works the same SSU2 or NTCP2

orignal do you even handle this situation?

zzz yes of course

orignal NTCP2?

zzz this is higher layer, at netdb layer

zzz transport doesn't matter

orignal you don't have peer test there

orignal and you basically don't need your peer's RI

zzz if you are going to lookup an RI and you are connected to the peer, just ask him

orignal no, i"m asking about peer test situation

orignal not lookup

orignal so that's what you are doing?

zzz no we don't do lookups for peer test

orignal how do you resolve it?

orignal If Alice's RI is not in your netdb anymore

zzz who am I again? Bob?

orignal I'm Bob, Alice sends peer test

zzz looking...

orignal through SSU2 session etsblished long time ago

zzz if (aliceRI == null) {

zzz if (_log.shouldLog(Log.WARN))

zzz _log.warn("No alice RI");

zzz // send reject

zzz sendRejectToAlice(SSU2Util.TEST_REJECT_BOB_UNSPEC, data, fromPeer);

zzz we just send unspecified failure code

orignal what is "unspecified code"?

zzz I guess we could ask Alice but we don't

zzz alice could send RI with peer test if session is up a long time

zzz but don't we send RI periodically now? I forget

zzz SSU2Util.java: public static final int TEST_REJECT_BOB_UNSPEC = 1;

orignal code 1?

zzz yes

orignal we send RI periodeically

orignal but it might be in between

orignal old expired new not sent

zzz shouldn't be

zzz we send RI every 29 minutes, and our shortest expiration is 60 minutes

orignal you create session with 5 mintes before expiration

orignal and send peer request after 15 minutes

zzz true

zzz but if peer is connected, we never expire without asking him for it first

zzz that's where we use the "direct lookup" code

orignal what do you mean?

zzz in our RI expiration code we have "lookup before dropping"

orignal so you remove a router from netdb only if it's not connected?

zzz try to get a new one, don't just delete it

zzz no

zzz first we ask him for a new one

orignal then explain

zzz if that fails then we delete it

orignal but it's netdb

orignal independent thing

orignal I go through netdb compare timestamps and delete expired

zzz right but you can send lookup messages directly to connected routers, even if not floodfill

orignal when?

zzz we do that, but then we 'lookup before deleting'

zzz any time

orignal it means I have to check in netdb

orignal if router is connected

zzz right, that's what we do

orignal overhead

orignal but fine

zzz up to you

orignal will try to do something

zzz protected void lookupBeforeDropping(Hash peer, RouterInfo info) {

zzz if (_context.commSystem().isEstablished(peer)) {

zzz // see DirectLookupJob

zzz ...

orignal Blinded message

zzz yeah we check the hash->connection maps in both transports

zzz so two quick hashtable lookups

orignal but for every router

zzz sure but expiration is a background job

zzz different thread

orignal but if you have one cpu no difference

zzz true ))

orignal I have better idea

orignal one you receive RI from SessionConfirmed bump it's timestamp

zzz yeah but that breaks the signature if you're ff and get asked for it

orignal I don't extract timestamp from buffer

orignal I store it seperately

zzz ok

orignal let me think

zzz we do about a bazillion hashtable lookups a second, if they are slow we're in trouble anyway ))

orignal but don

orignal 't you think

orignal that you are slow because of this ))

orignal for me every extra lookup slows down

zzz how often do you run the expiration code?

zzz most of the hashtable lookups are for configuration things: if (x >= _context.getProperty("max_xxxx")) ...

weko orignal: i2pd also have many hashtable lookups (as you said), but I guess it less count in many times))

orignal every minute

zzz orignal: too fast. Do RIs every 5 minutes. LS every minute.

orignal will do

orignal I save and cleanup at the time

zzz now you have the time to do connection check :)

orignal no I don't

orignal a router will be more busy ))

zzz ha

zzz we only save RIs to disk every 10 minutes

orignal even after start?

zzz yeah

orignal someone starts router then stop it after a minute

zzz so?

orignal and reseed again next time

zzz reseed writes directly to disk for us

orignal goof point

weko orignal: you did same, I am right?

orignal no

orignal I do it every minute

weko Write on disk?

weko I think we can do this configurable

weko And set default to 5 or 10 mins

orignal zzz also your opinion about garlic inside garlic. shoukd it be allowed or not?

zzz looking...

orignal I drop it for now

orignal but I think it's valid sitiation'

zzz what are the delivery instructions?

orignal local

zzz yes we allow it on receive. I'm not sure what the situation is when we send it though

orignal and handle?

zzz yes we handle it for LOCAL only. For DESTINATION we drop

orignal thanks. will implement it

zzz one case I know

zzz we send garlic'ed Delivery Status Messages with the LS to be returned back to us as an ack

zzz to hide the delivery status from the OBEP/IBGW

zzz afk, back in an hour

orignal no, it goes to destination not to router

dr|z3d orignal: if you're tweaking the way you save RIs, maybe consider not writing the crap to disk and keeping it session/memory only.

orignal too much memory

dr|z3d you don't store all your RIs in ram?

dr|z3d if you're reading from disk, then the alternative is to make L,M,N,U expire after 1hr and get deleted. keeps things from getting messy.

zzz the only thing we allow with delivery type DESTINATION is I2NP Data Message

zzz we would log anything else as an error

zzz never seen that error before

zzz if that was happening everybody would report it

orignal but it happened

orignal ofc you didn't see in the logs because it was local

orignal for router

zzz oh, I see, you were talking about Delivery Status Message. Never mind.

orignal I'm talking in general

orignal it was garlic in garlic sent to router

orignal either somebody's bug or attack

zzz tell us what's in it when you decrypt it

orignal I don't know

orignal I change the code to drop it

orignal because I had deadlock

orignal because recursion with mutex

orignal I need to redo this logic

orignal ppocess it instead drop

orignal and we have made the new release fixing this critical issue

zzz ok

zzz gosh DtQs is still at it. 7 inbound attempts in 9 minutes on 7 different IPs

zzz it's worth implementing a banlist for that one router alone

obscuratus Yeah, he's got a heck of a botnet (or something) going.

obscuratus They can't be connecting to that many Java I2P routers, they were banned by the newsfeed last month.

zzz i2pd + android _ bigly is still a lot

zzz i2pd + android + bigly

obscuratus I wonder how I2PD handles DtQs. Surely they can't connect to multiple DtQs at the same time.

zzz none of them have news feed blocking

zzz I would assume he would stop at one conn but who knows

obscuratus Bigly uses our lib without the newsfeed, How would we handle that if they weren't on the banlist.

zzz in ssu we drop the old one and switch to the new one

zzz not sure about ntcp

obscuratus It's got me scratching my head what they're trying to accomplish.

obscuratus I did a reverse DNS on a few of the DtQs IPs last week. They were all over the place, but included Cox, Comcast, and other typical providers of home service.

zzz I'm working on implementing an inbound throttle

zzz which is a headscratcher too