#i2p-dev (irc2p) | 2.12.0-8 | https://i2p.github.io/i2p.i2p/i2pupdate.zip | #translators for Weblate translation help

eyedeekay Hi zzz got time to chat about 538?

zzz lol that was fast

zzz havent looked at the diff yet, the CSS is messed up in-net and anubis is not happy on clearnet

snex delete anubis. install go-away

zzz but read your PR text a couple times

zzz what's the goal here?

eyedeekay Sorry I got disconnected, my home internet is all messed up, last thing I have is me asking "unhappy because..."

snex that never showed up

eyedeekay OK then: Unhappy because it's forcing you to re-validate too much or unhappy because it's buggy?

snex i guess you missed my suggestion too: "delete anubis. install go-away"

eyedeekay It's always on the list of possible solutions.

zzz back

zzz havent looked at the diff yet, the CSS is messed up in-net and anubis is not happy on clearnet

zzz but read your PR text a couple times

zzz what's the goal here?

dr|z3d If anubis or alternatives is the question, the goal is to eliminate ai bot/spiders.

eyedeekay regarding 538: my motivation here is in large part information-focused. I want to make it so we always know ahead-of-time:

eyedeekay - When a new stable version of a dependency is available

eyedeekay - Whether we can automatically update or some source of friction has been introduced(API changes, etc)

eyedeekay My hope is that we can use this to surface these changes sooner and respond to them faster and more incrementally. No high-pressure 8-version jumps.

eyedeekay In order to do that I made it so that updates could be applied automatically and tested in various contexts - right now just build tests but I was going to go over the unit tests and decide whether to run them too.

dr|z3d maybe I should read the PR :)

zzz ok

zzz imho this is pretty small potatoes

zzz don't share your goal of automated updates, that sounds bad

zzz don't think we've ever had a high-pressure jump before

zzz but more info is good

dr|z3d eyedeekay: have a look at the scripts/tools folder on the + repo. might give you some pointers for _some_ of the stuff requiring updates.

eyedeekay There's not any implied mandate to update the versions really, it's more about knowing what we're working with and surfacing the information

zzz dr|z3d, let us do this top-down before getting in detail weeds on impl

eyedeekay It will actually print you a nice little table

eyedeekay The automatic updates is mostly so the CI can generate the error report right now

zzz well, we know what we're working with, it's in LICENSE.txt, and it's basically my job to keep us up-to-date if necessary

zzz and all the build system stuff with with-xxx-java is also mine to maintain

dr|z3d minidns and simple-json were a bit behind the curve.

zzz yup, didn't say how good a job I _was_ doing, but it is my job

dr|z3d for current stuff, gmp you might want to update to 6.3.0, your gradle wrapper is a bit old, wrapper stuff, up to you if you want to keep that up to date.

dr|z3d *** chuckles. ***

zzz dr|z3d, _please_

eyedeekay TBH I didn't entirely know all the details myself, which is why I felt the need to surface the information, but because of that it's also entirely possible I overestimated the vendored library section

zzz well, it's all in my head and in LICENSE.txt, there's also a semi-maintained debian-alt/doc/dependencies.txt

zzz so maybe we need to automate it so that it's all in one place so you can understand it too, but maybe not, or maybe we take a look at if it's worth it

zzz eyedeekay, so on to your proposal, which is to use.... gradle???

eyedeekay Yeah that's where the plugins to talk to maven I used were, then the ant targets wrap the gradle scripts

eyedeekay \/wrap/call

zzz "plugins you used" ? remember I haven't seen the diff

eyedeekay Sorry there's an extended description in the PR too, I'll copy that part:

eyedeekay 'com.github.ben-manes:gradle-versions-plugin:0.54.0' - This is used for discovering the current(version we are using) and latest stable(Available on Maven) version of a dependency.

eyedeekay 'se.patrikerdes:gradle-use-latest-versions-plugin:0.2.19' - This is used for automatically updating the dependencies in the .gradle files.

zzz so this would be some new GH CI target?

eyedeekay Yes I have a first successful run of it here: github.com/eyedeekay/i2p.i2p/actions/runs/25703886733/job/75469769452

eyedeekay Demonstrating it catching a namespace change in the latest version of JSONSimple

zzz ok so there's two parts: 1) discovery and 2) auto-updating. let's take them one at a time

zzz where does the discovery output go to?

eyedeekay Github actions CI artifact is where it's going right now

zzz and who is monitoring that output? we have an ongoing discussion about GH CI that we haven't resolved, not sure if we're disagreeing or ignoring each other

eyedeekay I think we're mostly very busy, but I am now monitoring GH CI output much more carefully because it makes my life easier to do so, we're currently green

zzz do you get some notification on failure or do you have to remember to check?

eyedeekay I get emails, perhaps too many, but if I don't fix them my phone rings so much I call it the misery square

eyedeekay Because I get emails for every fork I follow that fails, which is not great, probably could filter better

eyedeekay But yes, I get notifications

eyedeekay Lots of them

zzz it's been years if ever since you told me 'hey you broke the CI today'. If notifications were working you'd be after me once a month or so

zzz so I don't buy it that our CI process is "working"

eyedeekay To some extent the problem is the noise, I do need to filter better especially when a sync failure doesn't resolve itself

eyedeekay But the last one to break CI was me on the 11th

zzz so that leads us to 2) auto-updating

zzz our process isn't set up well for it. nobody pays attention to GH PRs and issues that much, and having the CI auto-checkin would be a nightnmare

eyedeekay Oh I definitely don't think that sort of auto-update is defensible at all and I won't try to defend it, that's not my intent here

eyedeekay Auto-update as it stands is purely for discovering the compile errors and exposing them to humans

zzz because I always have to bug you to pull it back into gitea

zzz ok define auto-update then

zzz sounds to me like 'CI run with latest revs of things'

zzz right?

eyedeekay More-or-less that's exactly what it is, to tell us "Hey this thing changed with this library update so if you want to do the library update account for it"

zzz and that CI run is a gradle build? or you used gradle to grab stuff but then it's an ant build?

eyedeekay I use gradle to grab stuff and then it's an ant build

zzz that sounds... awkward

zzz as a summary, we have several types of 'vendoring' :

zzz 1) jars, opt-out for debian

zzz 2) source, partial or full, mods or not, opt-out for debian or not

zzz 3) source not in debian at all, but could probably be found in gradle from maven central

zzz there's no general-purpose way to do all the cases

eyedeekay I saw all those styles of vendoring while I did this. Mostly I knew enough to know I would have to account for debian so I found the flags and traced them through the buildsystem

eyedeekay One thing that I did not do was remove the source vendoring, there are new flags to trigger the jar-style vendoring for the CI builds but the source vendoring remains intact

eyedeekay Mostly because I didn't have time to review the mods/or not yet

eyedeekay \/flags/properties

eyedeekay There are also situations it can't surface yet either. Like the JSONSimple example, not safe to update to the one with the namespace change because the debian version is on the old namespace

zzz everything we support building without has a with-xxx flag listed in build.properties

zzz jsonsimple 3 is incompatibile with 2. Not sure what the story is with 4

eyedeekay From the looks of things, incompatible

zzz i2pcontrol and itoopie use it, maybe plugins too? so it's very messy

zzz not sure which of the numerous audit complaints about it are really in 2, I could only find one really

eyedeekay Re: flags those are the flags/properties I was tracing through the buildsystem to figure out how to make the PR

eyedeekay re: JSONSimple I only really learned of these issues by attempting to update it

zzz then of course there's stuff like jetty which can be a year-long project to upgrade major versions, CI's no help there

zzz we carry a patch from debian that's only used in the deb builds I think, that paches the source depending on whether you're building with 2 or 3

zzz this is from like 5 years ago

zzz they were trying to get all the packages off of 2 so they could get rid of it

zzz so I took it off debian and put it in our package

zzz see debian/patches/0003-json-simple-3.patch

zzz and grep for JSON in debian/rules

zzz yeah only for the true deb build, not in any of the debian-alts

zzz eyedeekay, can we shift to talk about gradle for a minute?

eyedeekay Sure

zzz so, gradle build is almost unmaintained

zzz and has no known users, except kinda is an intermediate step for android build somehow?

zzz I go through and fix up gradle about once a year

eyedeekay Accurate for Android, and I'll certainly buy that premise for users

zzz it goes months at a time broken in CI, as discussed above

zzz as you noted, we vendor a LOT of stuff into the gradle jars that should be dependencies, both stuff in debian and stuff that's not like minidns

zzz over the years I thought about fixing all that up, but no users and often broken and risks-breaking-android-build-but-dont-find-out-until-release

zzz -> not worth it, at least thats always been my conclusion

zzz if we had some in-house application that used gradle build that would be helpful, but that's not a great reason to do it

zzz but your 2) task ci-run-with-latest sounds 100x easier with gradle, just do the build again with the dependencies unpinned. Thats why I called using ant super-awkward

eyedeekay From the Android side it's sort of damned if you do damned if you don't. You can go without updating a long time but the longer you go the harder it is when something does eventually force a change that change turns out to be incompatible with all the other updates you've been avoiding

eyedeekay Yeah it would be a huge headache to express in ant XML

eyedeekay Oh wait no I'm misunderstanding let me read that back again

zzz you said you yanked the latest with gradle but then built with ant. I'm saying yank and build with gradle for the test

zzz and in the process the gradle build gets fixed up to de-vendor everything that should be

eyedeekay Sure yeah I can make it do that

zzz but it's fraught because no users and no good test application

zzz and risks breaking android but not finding out until too late

eyedeekay I can do a dev build and put it out for testing

zzz my one conclusion is that CI isn't a solution for any of our problems until CI processes are working well

zzz so I suggest lets put this stuff on hold

zzz 2nd conclusion on de-vendoring gradle, still not sure it's worth it, perhaps it's more important than new-version-discovery and should become the main goal

zzz 3rd conclusion re: dependency information and management, do we need to be better organized? should there be a chart somewhere, or in a different format, do you have one now, who should maintain it? (probably me)

eyedeekay I do have a chart now, and I think it could help to list them somewhere in a chart-like form

eyedeekay I could trim it down to just the gradle script called by `ant checkDepUpdates` which actually generates such a chart

zzz it's going to need to be manually maintained or at least the template will have to be

zzz how many line items you have?

zzz we have 13 with- flags, a few are obsolete

zzz there's about 15 listed in the dependencies.txt doc

eyedeekay 36 including everything, jetty, tomcat, jsonsimple, jstl, minidns and the new plugins which are also covered by the tool, but I'm listing sub-packages

eyedeekay *gradle plugins

zzz sounds like you have the best chart then

eyedeekay OK controlling for subpackages we agree, though, there are 15

zzz you really went down a rabbit hole, you've lived a nice life not having to worry about it ))

eyedeekay Well every once in a while I get a glimpse of an edge of it from Android

zzz this is why you need to stop and ask me for help faster when the launchpad builds break, you really went off in the weeds a couple weeks ago.

zzz this has all been on my shoulders since kytv vanished

eyedeekay Well even if the decision is that this is not a good idea at least I have a much deeper idea of what the buildsystem is actually doing

zzz sure so how about you review LICENSE.txt and debian-alt/doc/dependencies.txt for anything you missed, then put up your chart somewhere so we can look at it?

eyedeekay Sure I'll start with the PR thread

zzz ok pls check the in-net css

eyedeekay Ack will do shortly

zzz good discussion, gotta run, seeya

eyedeekay Thanks, see you later

dr|z3d I added some more unit tests, eyedeekay, not sure how useful unit tests actually are, but still, feel free to have a look and take anything that looks useful.

dr|z3d there's also an ant target that runs the tests and generates an html report.

eyedeekay We'll see. What would be most useful in the context we were discussing would be regression tests for specific behavior we rely on from earlier versions of libraries but that's highly specific and if we don't merge 538 no point

dr|z3d regression testing, good in theory, a nightmare to implement.

dr|z3d the easiest way to test for reressions is "does it compile after update?"

dr|z3d for libs, I'm doing a couple of different things, depending on whether they're dependencies or optional libraries for generating reports.

dr|z3d for dependencies, I now try to keep track of installed versions of, say, the wrapper with a version.txt file.

dr|z3d If I want to update to a new version, I just update to the desired new version in the version.txt and the build target does the rest.. pulls down the community pack, pulls down the source code for a win64 compile, and does the rest.

dr|z3d for optional report libs, I check for the latest version and update as required, then automatically update version.txt

dr|z3d not perfect, and not full coverage of all 3rd party libs/dependencies, but it makes thing a lot easier in places.