~dr|z3d
@RN
@T3s|4
@T3s|4_
@eyedeekay
@orignal
@postman
@zzz
%acetone
%mareki2p
%snex
+Atticfire
+FreefallHeavens
+Onn4l7h
+Onn4|7h
+fa
+marek22k
+onon_
+profetikla
+qend-irc2p
+r00tobo
+sexy
+uop23ip
Arch
Danny
Irc2PGuest21708
Irc2PGuest28384
Irc2PGuest4937
Irc2PGuest66257
Irc2PGuest75631
Irc2PGuest99986
Over1
RTP_
Watson
ahiru
anontor
cims
i2potus
interesting
justaperson
lokzz
luvme3
mahlay
makoto
n2_
nilbog
not_bob_afk2
pinotto
poriori
r00tobo[2]
rednode
sahil
uberius
user_ygg2_
zzz
dr|z3d, ping re: profiles
dr|z3d
zzz: what's up?
zzz
dr|z3d, struggling to deal with explosion of profiles on my ff, couple things I've tried so far not very effective
zzz
I see you have added a whole rats nest of stuff to profileorganizer, wondering how effective it is
zzz
how many profiles do you currently have on your ffs?
dr|z3d
~2400
zzz
I was at 19K
zzz
with some quick expiration tweaks I'm down to 10K
zzz
is that what your mods are about, just expiring things differently?
dr|z3d
there's some of that, but there's also a lot of "I don't think we care about this peer at all, let's just not bother with a profile".
zzz
that's where I'm headed
zzz
with my expiration tweaks I'm expiring 5000 profiles a minute, that seems like pointless thrashing
dr|z3d
yeah, my theory is we only really care about fast/high cap peers, and we definitely don't care about peers in low b/w tiers. not much to be gained from profiling slow peers we're not going to use for our own traffic.
zzz
can you point me to your 'we don't care' code please?
dr|z3d
as long as you promise to be gentle.
dr|z3d
*** grins. ***
zzz
I'll be polite ))
zzz
filename / method is fine, don't need GH links, I have the code here
dr|z3d
around line 596 in ProfileOrganizer is a good place to start.
zzz
but you're just using the output of that in reorganize(), to expire or delete things quicker along with the rest of your mods, right?
zzz
that code at least isn't preventing the profile from being created in the first place
zzz
you can be really aggressive in reorganize(), but doesn't that come at the cost of enormous thrashing?
dr|z3d
maybe somewhere else then, looking at ProfilePersistenceHelper.
zzz
that would just be at startup
zzz
so agreed with the 'we don't care' strategy, don't think where you're attempting to do it is efficient
dr|z3d
so we only save to disk after reorganize if the peers aren't already filtered out.
zzz
I tried it there and wasnt happy with results
zzz
disk or not, I'm expiring 5000 profiles a minute, you might be doing 4x that
zzz
if reorganize() is too late, then ProfilePersistenceHelper is WAY too late
dr|z3d
and you may be right, this might not be the most efficient method.
dr|z3d
are you seeing those 5000+ profiles reflected in the UI on /profiles ?
dr|z3d
I'm guessing you probably are, hence the pagination I think you added recently?
zzz
adding a local netdb lookup on each profile inside reorganize() isn't great, in theory the calculators should have done whatever was needed
zzz
I had 19K on /profiles
dr|z3d
ok, well, I do see a few lower tier peers on /profiles, but no real evidence of thrashing in terms of number displayed.
dr|z3d
I think our cutoff for display may be lower than what you're using, that also helps.
zzz
so tl;dr I started down the path that you did, but 100x simpler, just a couple lines, and I wasn't happy. YOu have a ** 9 ** step process in reorganize()
zzz
the pagination is on the 'all' tab, it had no limit, 19000 line table, browser wasnt happy
dr|z3d
haha, no, 19000 tables rows doesn't make for a happy browser, any which way you slice it :)
dr|z3d
currently all profiles shows around 1300 here. it might go higher, but probably not much higher than 2K. manageable in the browser.
zzz
ok thanks for the scoops. I have some ideas, I'm going to take a run at it from a different angle, will keep you appraised if it works out
dr|z3d
like, thanks for the insights. I'd like to cut slow peers off at the pass so they never get profiled in the first place, so maybe I'll look more into that.
dr|z3d
*likewise
zzz
or is it apprised? not sure
dr|z3d
the latter I think.
dr|z3d
apprise, to inform, update; appraise: to evaulate
zzz
ok will keep you apprised, and appraise your advice at $5
dr|z3d
now looking at bypassing profile creation in gProfileOrganizer.getOrCreateProfileNonblocking() and ProfileOrganizer.addProfile()
dr|z3d
$5, I'm rich! :)
zzz
that's roughly where I'm looking
dr|z3d
boolean isLowBandwidthTier(Hash peer) {
dr|z3d
RouterInfo peerInfo = _context.netDb().lookupRouterInfoLocally(peer);
dr|z3d
if (peerInfo == null) return false; // unknown, allow profile creation
dr|z3d
String tier = peerInfo.getBandwidthTier();
dr|z3d
return "K".equals(tier) || "L".equals(tier) || "M".equals(tier);
dr|z3d
}
dr|z3d
helper method, then just don't bother creating peers for anything that doesn't make the grade.
dr|z3d
*creating profiles
zzz
not taking code advice atm, and no I won't be adding more lookups
zzz
gathering data on call sites
dr|z3d
10-4, I'll leave you to it.
zzz
prelim data on which calls actually create new profiles:
zzz
3409 at net.i2p.router.peermanager.ProfileManagerImpl.messageReceived(ProfileManagerImpl.java:411)
zzz
778 at net.i2p.router.peermanager.ProfileManagerImpl.heardAbout(ProfileManagerImpl.java:343)
zzz
335 at net.i2p.router.peermanager.ProfileManagerImpl.messageSent(ProfileManagerImpl.java:35)
zzz
309 at net.i2p.router.peermanager.ProfileManagerImpl.dbLookupSuccessful(ProfileManagerImpl.java:189)
zzz
83 at net.i2p.router.peermanager.ProfileManagerImpl.dbLookupReply(ProfileManagerImpl.java:226)
zzz
6 at net.i2p.router.peermanager.ProfileManagerImpl.tunnelDataPushed(ProfileManagerImpl.java:140)
zzz
6 at net.i2p.router.peermanager.ProfileManagerImpl.tunnelDataPushed1m(ProfileManagerImpl.java:151)
dr|z3d
ok, some profile manager refinements, now slow peers never get profiled, and we have a 3 tier policy for retention. github.com/I2PPlus/i2pplus/commit/f97159358a0f26818ef1dc49fe5065546522b0c8
dr|z3d
may need tuning, will see how it performs.