#i2p-dev (irc2p) | 2.7.0-2 | Proposal 168 review Tues. Dec. 3, 8 PM UTC #ls2

zzz eyedeekay, looks like you had some fights with the news feed this week :)

eyedeekay Yeah then I realized that it pulls the article summary with a regular expression :P

zzz I don't know how you ended up putting the whole body of the post in the <summary>

zzz that won't work

zzz even if you did parse it right

zzz it wouldn't display right

eyedeekay Yeah and when I did it I realized I broke it and had to fix it which is what led me to find the summary thing

zzz the summary might not even be shown anywhere, or maybe just a tooltip

zzz don't spend more than 2 secs writing the summary

zzz looks like you got to the solution after several hours and several checkins

eyedeekay Yeah I was looking right past it, got the sections confused or something

zzz if it breaks, the problem is almost always your entries.html, not the generation scripts

lbt You guys really don't even have a name for that self-written web-site-builder/-server, do you? I still find that amazing/puzzling ;) I mean ... it's sounds a bit as you did an early Hugo as a by-product :P At least when considering that outputting a static version shouldn't be that difficult "what it does already"

dr|z3d that's a bit vague, lbt. what are you after? a static blog generator?

lbt I was comparing to Hugo and that is the obvious difference I am aware of

lbt But also ... if I were to host it, I would be after that feature indeed

dr|z3d still not sure what you're asking, but for static blog generators, look at lektor or bludit perhaps.

dr|z3d tor uses lektor for its blog, blog.incognet.io uses bludit.

dr|z3d (if you need some live examples)

lbt I'm looking into how things are done in this project. And kind of can't get over how little love/attention this part seems to have

dr|z3d oh, you're talking about the python scripts?

lbt Yes, lol

lbt See what I mean? ;)

dr|z3d yeah, there's a name. "python scripts" :)

lbt :))

eyedeekay If I ever have a kid I'll name them python scripts

dr|z3d no you won't!

eyedeekay j/k

dr|z3d if you ever have a dog, name it anarchy. then you can shout that everywhere.

dr|z3d :)

dr|z3d speaking of blogs, you ever get railroad updated, eyedeekay?

eyedeekay Couple time since we last talked about it IIRC

eyedeekay But the biggest remaining problem is that I'm building it on too new a system right now, I need to build it against an older glibc or stop using CGO

eyedeekay ideally both

dr|z3d ok, let's put that to the test. I fear the worst, but we'll see..

eyedeekay I doubt it's going to work unless you're using sid

dr|z3d sid? isn't sid fairly old?

eyedeekay sid's the unstable one, it's the very new one

eyedeekay Well, debian new

eyedeekay Not like, new new

dr|z3d I thought that was bookworm

eyedeekay No unstable is always sid, I think bookworm is testing now

eyedeekay sid's the neighbor kid who breaks your toys

eyedeekay (Toy Story references, a'la Debian)

dr|z3d sure, it's all toy story for whatever reason.

dr|z3d Failed to download plugin from idk.i2p/railroad/railroad-linux.su3

eyedeekay Hmm, let me check both the site homes, maybe the cron script failed again

dr|z3d you obviously got the memo about using monospaced fonts for body text.

eyedeekay Lol at least it's all on one stylesheet now, trying to get this behemoth of a site under control

eyedeekay But noted, I'll change it at some point

eyedeekay OK the script is running again, the railroad part should happen soon...

dr|z3d oh, and re sid, probably lunar lobster is more or less synced.

dr|z3d assuming railroad works, I don't know why you don't host idk.i2p using that.

eyedeekay Mostly the enormous incumbent collection of static content, moving it is the bit that sucks

dr|z3d I don't know how railroad functions, but if it's anything like bludit, you'd just strip the <html> <head> and <body> tags and paste the main content html straight into a post. then worry about the top menu / navigation afterwards.

eyedeekay Oh I see what's wrong, the version number is configured incorrectly in the script that syncs the sites

eyedeekay I'm getting rid of this CGO bullshit while it's on my mind and since I've got go-i2pbrowser now

eyedeekay There, freakin' finally, no more glibc on linux builds

eyedeekay Forgot I could do that now

lbt Do I understand correctly that "advanced search" (premium whatever) is required to do a search with like "-filename:*.po"?

lbt Talking GitLab here, sry

dr|z3d lbt: you got the repo download locally?

lbt Ya, using freemiumg grep -v now :o

dr|z3d lbt: if so, to retrieve a list of all .po files in the workspace, you can run: git grep --files-with-matches .po |grep ".po$"

lbt Thing was that the po files were cluttering the search results so I wanted to exclude them from searching in the files basically

dr|z3d git grep != grep

dr|z3d git grep is about a million times faster.

lbt Oh, interesting :)

dr|z3d run it in combination with standard grep to drill down on your search.

lbt Will check. On this new machine everything seems lightning fast anyways though :o nvme-ssd kinda rocks I guess, especially when you are not used to it yet ;)

dr|z3d eg git grep "foo" |grep --exclude=bar

dr|z3d or whatever.

dr|z3d git grep uses the git db, so it's not running a live search. which is why it's much faster.

lbt That would still grep all files though, or? Not that it matters, but find | grep -v | xargs grep filters out first. Well, git-grep sounds like worth checking out anways :)

dr|z3d yeah, all files in the workspace afaik.

dr|z3d or maybe all committed/added files. you'd have to double check that.

lbt Need to understand more of git anyways. Messed up a bit earlier

dr|z3d very handy for finding strings in source code.

dr|z3d eg git grep "PROP_"

dr|z3d git's easy to mess up, and once you've got a few key skills under your belt, almost as easy to recover from.

lbt It's actually what's it's there for even. I just need to understand some more basic concepts (the trees) and things will start to fall together (more) I think

dr|z3d often useful is the "git checkout ." command, if you mess up your local workspace and want it returned to the state of the last commit (aka HEAD).

dr|z3d a tree is just the source directory structure. you probably meant branches :)

lbt Oh really? I ended up doing some ... was it reset HEAD~ or so? And then all of sudden 2 commits from zzz were unstaged there and I went "ooooh ..."

dr|z3d git stash and git pop also useful.

lbt Ah, there is worktrees or something? But ya, not understanding underlying concepts already sounds bad ...

dr|z3d (if you've got a locally modified workspace and want to pull from a repo)

dr|z3d git reset ~HEAD will more or less achieve the same thing.

lbt Or even just wording, if tree = worktree :P

dr|z3d probably

dr|z3d sounds like tree, worktree and workspace more or less refer to the same thing.

lbt Ya maybe I had done the pull upstream inbetween or something. checkout . definately is noted for the next time ;)

dr|z3d if you don't want to lose your changes, git stash before git pull.

dr|z3d then you can git pop

dr|z3d (after pull)

dr|z3d git checkout <filename> will restore an individual file to it's committed state, and you can use wildcard syntax to hit more than one file.

dr|z3d eg git checkout **/console.css

dr|z3d that would restore any modified file named console.css to its committed state.

lbt That's all on uncommitted changes though, right? I had a local commit that I wanted to refactor - then noticed I'm on master, not in a branch. Oh well, I guess you've heard that kind of story before ...;)

dr|z3d still, if you want to keep your main branch clean and pull from that without issue, just git checkout -b <newbranch> for your local work

lbt That's what I had forgotten there and only noticed when I was about to push.

dr|z3d yeah, you want to work on your own branch and then you can commit that to your branch and then merge it with the main branch, cherry-pick the specific commit, or push it to a separate branch and issue a merge request from there.

dr|z3d obviously to switch between branches you just do git checkout branchname. to see available branches and your current branch, git branch

lbt branches are supposed to get deleted after a merge then via some gitlab "allow this" magic, right?

dr|z3d deletion is optional.

lbt So if I don't select that, I could re-use a branch (name) to work on the same topic further?

dr|z3d more a matter of taste and working style than anything. do you want multiple branches available on gitlab, or do you want to develop locally in a different branch, push it to the remote repo and then merge with main/master and then delete it? up to you.

lbt Deleting/not cluttering doesn't sound bad actually

dr|z3d I do all my work in a branch named dev, with an upstream branch for upstream pulls. when I push to dev, as soon as the commits are merged with master (on gitlab), dev branch is nuked.

dr|z3d on github, I leave the dev branches there, since github seems to want the entire dev branch uploaded with every commit if the dev branch isn't already there.

lbt Hm, you do not have a forkin there but are working directly with the i2p.i2p repo, right?

dr|z3d I have a local dev branch and an upstream branch, upstream gets pulled and then, working in my dev branch, commits get cherry-picked as required.

lbt Ya. I have a fork (origin) and then local + upstream. One thing more to make a mistake (like pulling my origin instead of upstream when preparing a merge request)

dr|z3d given the amount of divergence, I can't just pull from upstream into my dev branch.

dr|z3d so I cherry-pick instead.

lbt Why are you so divergent? ;) Like many (larger) projects that changed lots but didn't finish and now "temporarily don't update"? Is it something like that?

dr|z3d no

dr|z3d just a huge amount of visual and non-visual differences.

dr|z3d you'd have to run I2P+ to really see what I mean :)

lbt I haven't even gotten as far as checking what it is albeit seing the name :( I'm good in getting lost in details ;)

dr|z3d :)

dr|z3d if you're curious, skank.i2p

lbt Much too curious. Don't you trigger me ;)

dr|z3d well done, eyedeekay, railroad now works.

lbt Hm, easier to use, better performing, same security. You got some marketing going :)

dr|z3d :)

dr|z3d let me know when I2P supports up to 1Gb/s bandwidth share.

dr|z3d eyedeekay: I say works, the homepage at least loads now, any other page, though, slq error.

dr|z3d *sql

dr|z3d ah, the default pages appear to be placeholder links without content is probably why.

lbt I didn't know there is so much traffic around ;) My routers seem to never reach limits - memory usually goes steadily up till garbage collection (I guess), but other than that there always seems to be free bandwidth for grabs basically

dr|z3d mostly there isn't, except when the swarm of potentially abusive routers gets going. then I've seen 42MB/s

lbt Hm, how easy is it to get detailed router stats for monitoring? "icinga-plugin where?" ;)

dr|z3d > /configstats is your best bet.

dr|z3d enable advanced stats and away you go.

dr|z3d eyedeekay: DSA_SHA1 sigtype?!

lbt Damn, should have done something there 10y ago. Would be interesting to have a look at now. I don't even know how reachable my blog has been :o

dr|z3d eyedeekay: also, railroad's not correctly registering with the console plugin manager .. indicates it's stopped after autostarting.

dr|z3d ok, seems to work after a start/stop.

lbt btw latest I2P version: 2.0.0 - #i2p.chat

dr|z3d you'd have to mention it there, different channel, different ops.

eyedeekay dr|zed yeah I've been working my through a bunch of stuff here, going to release a new version in a little while

dr|z3d ok, eyedeekay, thanks. at least it actually works now \o/

dr|z3d you probably also want to set the name somewhere so it doesn't appear as "SAM MUX host" or whatever.

dr|z3d default page is also using google's font api, probably worth neutering that.

dr|z3d it might also be worth the effort to consolidate the difference OS versions into a single plugin and then just check the OS and run the appropriate binary.

dr|z3d (or maybe the go runtime a dependency plugin and download that separately)

dr|z3d *make

dr|z3d when I need a distraction I might take a deeper dive into the code and see what can be improved presentation-wise.

dr|z3d plugin description as appears on /configplugins needs to be a lot briefer. help/instructions for deployment in the README, not the description!

dr|z3d eg "Personal Blogging platform"

lbt Have timestamps for i2psnark (the log thing on top) ever been suggested/discussed?

eche|off state report: main system broke down, new hardware ordered, guess until sunday I'll be mostly offline

zzz :(

eche|off yeah, unexpected

eche|off but still reachable on twitter and mail

zzz any services down?

eche|off no

eche|off not the root servers

zzz ok

eche|off just my personal private system at home

eche|off but thats the system I used to manage most time

eche|off sorry, update to 2.1 donw, after seeing wild spikes from 200k to 4 MB/sec during 1h timeframe

eche|off inbound limit hit, hmmm

zzz yup, wild ride started about 24h ago, awaiting bigly update

eche|off yeah

eche|off but graphs do show for e.g. 1h time 2 MB/sec out and 4 MB/Sec in

eche|off on a router with default services, which were mostly even distribute

eche|off so it is far to much in traffic which did hit the limtis

zzz that's fine, limits are good

eche|off sure, but part tunnel count goes to null after that

eche|off or "near zero"

eche|off router active peers go down, to

eche|off so limits DO work :-)

zzz sure, when limits are hit, we do things to fix it, but shouldn't go to near zero, unless only a few tunnels are hogging all your bw

zzz check /tunnels

eche|off na, that was nearly 20h ago, no realtime data

eche|off 3577415183RU gw5o3058214896RU 37-94 Minuten291 MiB918 KBpsTeilnehmer although that one is nice

zzz ok plus standalone, a little janky but not bad dr|z3d

zzz why in the world are you doing a XHR for all.css every 5 sec?

zzz i think bigly flipped their switch about 8 hours ago

obscuratus I have OutboundEstablishState and OutboundEstablishState2 on Debug log level. A lot of my 'udp.OutboundEstablishState2: new outbound establish to' messages will fill in the 'with address:' portion with ssu://autodetect.not.yet.complete:0, or maybe I have one good ssu address, and the rest look like: ssu://46005023@null:0

zzz get too close to the TV, all you see is spots

dr|z3d zzz: build success readout in the sidebar. any more thoughts? :)

zzz our trend is toward less stuff there, not more

zzz and nothing the user can do about it

zzz got a report '5 tunnels using 90% of my traffic', haven't seen that myself, but may be worth a review

dr|z3d true, there's nothing the user can do to make the build success better, but it's a good indicator of overall network health.

zzz actually it's not, it's a local view of health. I'd rather they go to stats.i2p for the overall view

zzz reddit and ilita are filled with q's about what they can do about i2pd build success, let's avoid that

dr|z3d nothing a post can't remedy. but I take your point. in which case, put it in the congestion section that's only enabled by default in advanced mode :)

zzz I never look at my own, it's just not useful

zzz imho if I don't need it nobody else does, but if you disagree, have fun, % = success / (success + reject + timeout)

dr|z3d lol

dr|z3d that's an amusing criteria, but thanks.

dr|z3d re tunnels, I've seen a small proportion using an oversized amount of b/w before, prior to us banning abusive routers.

dr|z3d 5-10 tunnels pulling 2MB/s each. something like that.

obscuratus dr|z3d: I've been running this patch for a while now. It makes a graph of your local exploratory build success.

obscuratus paste.idk.i2p/ChaetognathanColt/read

obscuratus One drawback, it gives a number between 0-1000. I couldn't figure out how to store floating point numbers.

dr|z3d yeah, but does it provide the results as a percentage? :)

zzz also as soon as you put stats in front of people they start complaining you didn't smooth it right, and you get bombastic PRs like this one github.com/PurpleI2P/i2pd/pull/1833

dr|z3d haha

dr|z3d yeah, that's true, zzz. always gonna be complainers. armchair critics.

dr|z3d obscuratus: stats are stored as a double afaik. in which case, to get an int percentage, you'd do something like (Math.toIntExact(statvalue) / 2)

dr|z3d * 100 sorry.

dr|z3d no, / 10.

dr|z3d if max value is 1000.

dr|z3d let's see that PR request.

dr|z3d oh greay, all in russian. looks good though, pictures and everything.

dr|z3d *great

obscuratus dr|z3d: Maybe it's out there and I couldn't find it, but I haven't run across a stat that stores in floating point. Can you point me to an example?

dr|z3d obscuratus: stats are stored as a double. I;m saying for a percentage, an int value is fine.

zzz the issue with build stats is you have to take the event counts, not the average value, which is a resp. time

zzz so you get the event counts for all three and then do the math I said above ^^^

obscuratus Yeah, I use the event count that we already store.

dr|z3d buildHealth = (1000 * expSuccessEvents) / (expSuccessEvents + expExpireEvents + expRejectEvents);

zzz and if you wanted to graph it you'd have to make another "derived" stat and update that periodically

obscuratus zzz: exactly.

dr|z3d looks like that patch is heading in the right direction, obscuratus, but ideally we want build success for all tunnels, not just exploratory.

zzz as drz says everything is a double. if you don't like the scale then change it on the way in

zzz you probably want to multiply by 100, not 1000 :)

dr|z3d yeah, I was hinting at that. percentage is a better metric. :)

zzz mystery solved

obscuratus dr|z3d: Feel free to change that around any way you see fit (if you're interested). I thought I'd throw it out there, but it's not like there's a lot to it.

dr|z3d thanks, obscuratus, I might see if I can get a nice percentage value for all tunnel builds. feel free to attempt the same in parallel :)

zzz obscuratus, change from 1000 to 100d, that will force the rest of the computation to double and not truncate

zzz and declare buildHealth double

obscuratus zzz: Thanks, I'll give that a look.

dr|z3d obscuratus: if you want to test adding it to the sidebar, modify the following code to suit and then insert into renderCongestionHTML() on SummaryBarRenderer,java .. the config property is optional, obviously.

dr|z3d if (_context.getProperty("routerconsole.showPeerTestAvg") != null &&

dr|z3d _context.getBooleanProperty("routerconsole.showPeerTestAvg")) {

dr|z3d buf.append("<tr title=\"")

dr|z3d .append(_t("Average time to test a peer (successful / total)"))

dr|z3d .append("\">" +

dr|z3d "<td><a href=\"/profiles\"><b>")

dr|z3d .append(_t("Peer test average"))

dr|z3d .append("</b></a></td><td class=\"digits\"><span>")

dr|z3d .append(_helper.getAvgPeerTestTimeGood()).append(" / " )

dr|z3d .append(_helper.getAvgPeerTestTime());

dr|z3d if (_helper.getAvgPeerTestTime() > 0)

dr|z3d buf.append(" ms");

dr|z3d buf.append("</span></td></tr>\n");

dr|z3d }

obscuratus dr|z3d: Thanks.

zzz dr|z3d, 1st line not needed, null == false

dr|z3d ok, zzz, thanks.

zzz DataHelper.formatDuration2() might also help for the times

dr|z3d yeah, maybe, though in this instance ms is fine.

dr|z3d formatDuration2() would start using seconds and floating points, which isn't what I want there.

dr|z3d the leading value is the most important, should always be ms.

zzz ok

zzz harder if you want "a / b ms" esp. if the a and b scales could be different

dr|z3d yeah. that's why a static "ms" is fine there. values vary from 300 - 2000 ms, so it's all good.

dr|z3d plus, there's an override config that sets the peer test timeout, in ms, so having values reported there in anything else could be confusing.

zzz see formatPair() where we coerce a and b to the same scales

dr|z3d yeah, gotcha, thanks. works for b/w readouts.

zzz annoying to get right

dr|z3d yeah, I bet, but satisfying once you do :)

dr|z3d speaking of smoothing, I'm currently taking a realtime value / 100, plus a 1m avg / 100, dividing both by 2 and then adding to get a something smoothed value. sane?

dr|z3d not super important to be 100% accurate. seems to be working fine enough.

dr|z3d using the 1m average helps to prevent the value from jumping all over the place, given that it drives a percent bar.

zzz I would just pick a rate and use that alone, don't need anything fancy, we don't do that anywhere else

dr|z3d yeah, thing is, 1m is too long to give a responsive readout. that's the first thing.

dr|z3d and all my graphs are standardized over a minute, so setting the rate differently would just screw up the graphs :)

dr|z3d or can you set multiple, out of order rates for the graphs and always have it use the first one?

dr|z3d {1m, 10s, ...}

dr|z3d cpu load averaged over a minute is not particularly useful, hence the mixin with the realtime readout.

zzz i think it's 1 minite minimum

zzz how about (currentTotalValue + lastTotalValue) / (currentEventCount + lastEventCount) ? that would give you rapidly changing smooted values

zzz or is that what you just said

zzz yeah my way weights the two buckets correctly

zzz if you want a little decay:

zzz how about ((2*currentTotalValue) + lastTotalValue) / ((2*currentEventCount) + lastEventCount)

dr|z3d I can try that, see if gives me superior results. thanks. if you want to try for yourself, code's up on gitlab.

zzz your way vastly overweights a baby current bucket with only a couple samples

dr|z3d if the stat value = 0, it'll just use the current value instead.

zzz yeah but if the current event count is 1 and the last event count is 100, you're overweighting the current bucket by 100x

zzz so thats the opposite of smoothing :)

dr|z3d ok, I knew it wasn't totally accurate, I'll try your method. the main thing is to reflect cpu load spikes while giving a reasonable average.

dr|z3d :)

dr|z3d gitlab.com/i2pplus/I2P.Plus/-/blob/bbc8a9d091632633d115c2ec6dfb87b893301707/apps/routerconsole/java/src/net/i2p/router/web/helpers/SummaryHelper.java#L501

dr|z3d Rate stat = _context.statManager().getRate("router.cpuLoad").getRate(60*1000);

dr|z3d long loadAvg = (long) ((2*stat.getCurrentTotalValue() + stat.getLastTotalValue()) / (2*stat.getCurrentEventCount() + stat.getLastEventCount()));

zzz another sharp "health" spike up but after my premature call saturday night I'm remaining cautious

zzz yeah thats it dr|z3d, see how it goes, that over a 60-sec rate should get you constant smoothed changes in the sidebar and a nice graph

dr|z3d thanks, zzz, I'll keep an eye on it.

dr|z3d the graph is as before, just averaging the value over a minute.

zzz just calculate and check for zero summed event count so you dont explode

dr|z3d not sufficient? int load = loadAvg != 0 ? (Math.toIntExact(loadAvg) / 2) + (getCPULoad() / 2) : getCPULoad();

zzz should be sufficient

dr|z3d great, thanks. it just falls back to a realtime value if the stats are returning 0.

dr|z3d what's just as interesting is the potential to throttle or otherwise modify various sub-systems based on the reported cpu load. helps keep the router from not killing the cpu.

dr|z3d just noticed, I'm still feeding in the realtime value as 50% of the overall value.

zzz if it were me I'd explicitly calculate and check the denominator before doing the division to be bulletproof, a divide-by-zero is an unchecked exception

zzz so you can't do a = b / c; if (a == 0) .... // too late

dr|z3d that's whay the above does.. checks the value isn't zero before doing the division.

dr|z3d or not?

zzz you need

zzz c = d + e

zzz if (c > 0)

zzz a = b / c

zzz else

zzz a = getcpuload()

dr|z3d gotcha.

dr|z3d long loadAvg;

dr|z3d if ((stat.getCurrentTotalValue() + stat.getLastTotalValue()) > 0 && (stat.getCurrentEventCount() + stat.getLastEventCount()) > 0)

dr|z3d loadAvg = (long) ((2*stat.getCurrentTotalValue() + stat.getLastTotalValue()) / (2*stat.getCurrentEventCount() + stat.getLastEventCount()));

dr|z3d else {

dr|z3d loadAvg = getCPULoad();

dr|z3d }

zzz yes, that's _almost_ bulletproof

zzz because the event counts could get zeroed between calls

zzz so only call once

zzz long count = (2 * currentEventCount() + lastEventCount();

zzz if (count > 0)

zzz loadavg = ((2*currentTotalValue()) + lastTotalValue()) / count

zzz _thats_ bulletproof

dr|z3d alrighty, thanks. :)

dr|z3d I've added whatever java itself thinks the cpu load is as part of the calculation, zzz. seems to work ok.

dr|z3d long loadAvg;

dr|z3d long count = (1 + (3 * stat.getCurrentEventCount() + stat.getLastEventCount()));

dr|z3d if (count > 1) {

dr|z3d loadAvg = (long) (getCPULoad() + ((3 * stat.getCurrentTotalValue()) + stat.getLastTotalValue()) / count);

dr|z3d } else {

dr|z3d loadAvg = getCPULoad();

dr|z3d }

dr|z3d int load = Math.toIntExact(loadAvg);

dr|z3d and bumped up the current/previous ratio.