#i2p-dev (irc2p) | dev build 2.5.0-4 | please test!

zzz ping eyedeekay re: website

eyedeekay I'm here, Which thing zzz?

zzz eyedeekay, you apparently broke almost every single translation on the entire website 5 months ago, unnoticed until now

zzz your march 30 checkin looks like you did a tx pull but not a update-existing-po.sh, which breaks all multiline translations

zzz also, you've subsequently checked in some translations, uncommented, with e.g. meeting logs, not sure why

zzz I pulled the new translations and checked them in, that fixed those... then I reran update-existing.po and checked it in, should be live in a few minutes

zzz please don't combine translation updates with other checkins

zzz please be very careful to follow the instructions in the readme when updating translations

zzz it's really bad to have our SEO and translations completely broken for almost 5 months

eyedeekay Oh my gosh I'm so sorry, I didn't realize that was what was going on but I remember exactly when what you're talking about happened

eyedeekay Had an email session with echelon about it and everything, I'll re-read everything right away and ping you about questions

zzz well, if you knew you broke it and never fixed it, that's even worse :(((

eyedeekay If I have anyway

eyedeekay *any

zzz I'm also updating the source files and pushing them to tx, we haven't done that in almost a year

eyedeekay No I didn't realize it was broken I just remember the time when I accidentally broke it

zzz ok. and why are translations getting checked in with unrelated stuff like meeting logs?

eyedeekay Same as when I made the same mistake in other places, I was in a hurry and didn't separate the checkins

eyedeekay I made that mistake a lot about that time, trying to do better

zzz ok. I don't think echelon knows much about the website translation stuff, that was all str4d wizbangery. I've been doing the push/pull for years, but not often enough

zzz just be careful, follow the readme, and take your time if you're going to do it

eyedeekay Well I'll re-read it and get it right next time so we have some backup

eyedeekay Will do, sorry for the mistake, thanks for letting me know about it

eyedeekay Can't improve if I don't know

zzz yup

zzz I assume you're working on android this weekend?

eyedeekay Android and also testing Firefox Profile+Separate I2P Router scenarios

zzz don't forget there's some threads on my forum with minor android todo's in there for you, can't remember what or where

eyedeekay Yeah text changes mostly IIRC, I know where they are

eyedeekay Oh and the b32 validator

zzz again, take your time, test and be careful, let's have a good release, when it's ready and not before

eyedeekay Will do

zzz eyedeekay, another thing for your list, after the release...

eyedeekay lay it on me

zzz we'd like to make a list of user count by language, so we can make a prioritized list of the translation "gaps" and give that to the Loc. Lab

zzz you should be able to get that from the ?lang param in the news server logs

zzz this will be better data than just the by-country geoip info from the console

eyedeekay Yeah they will have selected it on purpose, makes perfect sense

zzz I assume eche's news server is logging requests

zzz well, yeah, on purpose if they went thru the wizard on first install, but we'll get it from the locale if not

eyedeekay Ech's server does log requests, mine does too but they go away when I rebuild my news, so my sample will only be for one update cycle for instance

eyedeekay shouldn't be a problem to come up with a count

zzz ok

zzz eyedeekay, also, eche's i2pnews.i2p is returning 503, please investigate

HaruCode i2pforum.i2p does the same. probably the same server

eyedeekay OK I'll figure out what's going on

zzz thanks, if you're stumped lmk and I'll DM eche

eyedeekay Woo got at least part of it

eyedeekay i2pforum clearnet side is back up

HaruCode indeed. out of storage space due to logs?

eyedeekay Maybe, I just got back into it, still figuring out what happened

eyedeekay Close, old backups

eyedeekay And also logs

eyedeekay I2P side is back up too

HaruCode the usual thing whe something works, works, works and then suddenly 503. things need maintenance, you know

HaruCode *when

eyedeekay Worrying part was that I lost SSH access when it happened too, had to fix it from the VM host, higher stakes

HaruCode whether it is worrying, depends. some system can't accept new ssh sessions when out of storage space. if you can login without changing anything after you freed the space, it's probably ok

eyedeekay Re: i2pnews it's not the disk, it's not the server, and it's not the feed so it's got to be the reverse proxy

HaruCode to me it returns 200 and shows two broken graphs

HaruCode the graphs do return 503 though

HaruCode so it'ts the script which draws the graphs, unless you see no error in direct requests

eyedeekay No issue on direct requests here, that's why I think what I think for now

eyedeekay Might find out different

HaruCode if the backend server for the page itself and the graphs is the same, the database connection might be the issue

HaruCode oh. if it's ok in direct requests, then it's probably proxy indeed. try cleaning its cache

HaruCode also hostnames might be the problem

HaruCode you might see no error because you send a different hostname in your request

HaruCode or it's your result that is cached and the actual state is bad

eyedeekay Could be

eyedeekay not the state, making my requests with wget, should be no cache

HaruCode free space or config on proxy then

HaruCode can't say for sure without looking. try tracing the traffic between the proxy and backend

eyedeekay Proxy is trying to talk to the wrong hostname, but since it worked before and didn't change, and I can still reach it from the proxy host with both wget and curl, so still some kind of mystery to be solved

HaruCode not quite get it: what hasn't changed and what can be reached? and if you didn't change something it doesn't mean other admin didn't. if there are other admins, of course

HaruCode and the wrong hostname might me not related to this site. or it's a bad sign meaning the config has been tampered with. depends on whether you know this "wrong" hostname

eyedeekay Proxy trying to reach the same hostname as it always has, but it isn't able to. But, if I log in to the proxy host and try the same hostname with curl, it works just fine

eyedeekay Looking for clues in the config files

HaruCode aha. local dns issues are likely

HaruCode try using tcpdump to compare how the proxy and curl resolve the name

HaruCode and maybe grepping the configs for the resulting ip

HaruCode and if the name resolution is correct, dump the traffic to see differences in http streams

eyedeekay OK got the name resolution fixed, according to the logs all requests for su3 files have been successful for about 10 minutes and no more failures to connect to the news host, but it's still 503'ing on the homepage

dr|z3d if it's nginx, check /var/log/nginx/error.log or wherever it's stored, might shed some light.

eyedeekay It's apache2 unfortunately, I'm much better at nginx but I'll get it

HaruCode http-error.log should help

eyedeekay I'm looking around in the logs for clues too

HaruCode if it's apache, necessary modules might have not been loaded

HaruCode of the dependencies for the modlue used, like php or perl additional modules

eyedeekay Only new information I've got so far is a warning "Flask-Cache: CACHE_TYPE is set to null"

HaruCode that only means the internal cache is disabled

eyedeekay Yeah it seemed unrelated when I stuck it in a search engine

HaruCode python modules and updates can be tricky. maybe an update happened that broke it

HaruCode the http(d)-error.log has no details?

HaruCode usually the problem is unintended python version switch

eyedeekay Nothing new in the httpd error logs unfortunately

HaruCode well, reinstalling the related packages or rebuilding from sources might reveal the problem, but I can't say how exactly it should be done, since it's os-dependent

eyedeekay Yeah I'm always breaking python stuff that way, gotta be super-careful

HaruCode check which python versions you have and which is the default. the wrong python is number 1 in the list of error caues

HaruCode *causes

eyedeekay about to be sick, back in ~20

eyedeekay That sucked

StormyCloud eyedeekay: You all good?

eyedeekay Yeah I'll be fine

eyedeekay Hey that's a little more helpful, 429 denied, access-control-allow-origin

eyedeekay Got it to show me something useful, pretty sure if I fix that we're back to normal...

eyedeekay Nope, I was wrong, fixed the 503's but the 429's are still there

eyedeekay Nope, wrong about that too

eyedeekay Damn

eyedeekay and yet, every single su3 download seems to work without incident

eyedeekay It is down to just the script that generates the stats svgs