zzz
ping eyedeekay re: website
eyedeekay
I'm here, Which thing zzz?
zzz
eyedeekay, you apparently broke almost every single translation on the entire website 5 months ago, unnoticed until now
zzz
your march 30 checkin looks like you did a tx pull but not a update-existing-po.sh, which breaks all multiline translations
zzz
also, you've subsequently checked in some translations, uncommented, with e.g. meeting logs, not sure why
zzz
I pulled the new translations and checked them in, that fixed those... then I reran update-existing.po and checked it in, should be live in a few minutes
zzz
please don't combine translation updates with other checkins
zzz
please be very careful to follow the instructions in the readme when updating translations
zzz
it's really bad to have our SEO and translations completely broken for almost 5 months
eyedeekay
Oh my gosh I'm so sorry, I didn't realize that was what was going on but I remember exactly when what you're talking about happened
eyedeekay
Had an email session with echelon about it and everything, I'll re-read everything right away and ping you about questions
zzz
well, if you knew you broke it and never fixed it, that's even worse :(((
eyedeekay
If I have anyway
eyedeekay
*any
zzz
I'm also updating the source files and pushing them to tx, we haven't done that in almost a year
eyedeekay
No I didn't realize it was broken I just remember the time when I accidentally broke it
zzz
ok. and why are translations getting checked in with unrelated stuff like meeting logs?
eyedeekay
Same as when I made the same mistake in other places, I was in a hurry and didn't separate the checkins
eyedeekay
I made that mistake a lot about that time, trying to do better
zzz
ok. I don't think echelon knows much about the website translation stuff, that was all str4d wizbangery. I've been doing the push/pull for years, but not often enough
zzz
just be careful, follow the readme, and take your time if you're going to do it
eyedeekay
Well I'll re-read it and get it right next time so we have some backup
eyedeekay
Will do, sorry for the mistake, thanks for letting me know about it
eyedeekay
Can't improve if I don't know
zzz
yup
zzz
I assume you're working on android this weekend?
eyedeekay
Android and also testing Firefox Profile+Separate I2P Router scenarios
zzz
don't forget there's some threads on my forum with minor android todo's in there for you, can't remember what or where
eyedeekay
Yeah text changes mostly IIRC, I know where they are
eyedeekay
Oh and the b32 validator
zzz
again, take your time, test and be careful, let's have a good release, when it's ready and not before
eyedeekay
Will do
zzz
eyedeekay, another thing for your list, after the release...
eyedeekay
lay it on me
zzz
we'd like to make a list of user count by language, so we can make a prioritized list of the translation "gaps" and give that to the Loc. Lab
zzz
you should be able to get that from the ?lang param in the news server logs
zzz
this will be better data than just the by-country geoip info from the console
eyedeekay
Yeah they will have selected it on purpose, makes perfect sense
zzz
I assume eche's news server is logging requests
zzz
well, yeah, on purpose if they went thru the wizard on first install, but we'll get it from the locale if not
eyedeekay
Ech's server does log requests, mine does too but they go away when I rebuild my news, so my sample will only be for one update cycle for instance
eyedeekay
shouldn't be a problem to come up with a count
zzz
ok
zzz
eyedeekay, also, eche's i2pnews.i2p is returning 503, please investigate
HaruCode
i2pforum.i2p does the same. probably the same server
eyedeekay
OK I'll figure out what's going on
zzz
thanks, if you're stumped lmk and I'll DM eche
eyedeekay
Woo got at least part of it
eyedeekay
i2pforum clearnet side is back up
HaruCode
indeed. out of storage space due to logs?
eyedeekay
Maybe, I just got back into it, still figuring out what happened
eyedeekay
Close, old backups
eyedeekay
And also logs
eyedeekay
I2P side is back up too
HaruCode
the usual thing whe something works, works, works and then suddenly 503. things need maintenance, you know
HaruCode
*when
eyedeekay
Worrying part was that I lost SSH access when it happened too, had to fix it from the VM host, higher stakes
HaruCode
whether it is worrying, depends. some system can't accept new ssh sessions when out of storage space. if you can login without changing anything after you freed the space, it's probably ok
eyedeekay
Re: i2pnews it's not the disk, it's not the server, and it's not the feed so it's got to be the reverse proxy
HaruCode
to me it returns 200 and shows two broken graphs
HaruCode
the graphs do return 503 though
HaruCode
so it'ts the script which draws the graphs, unless you see no error in direct requests
eyedeekay
No issue on direct requests here, that's why I think what I think for now
eyedeekay
Might find out different
HaruCode
if the backend server for the page itself and the graphs is the same, the database connection might be the issue
HaruCode
oh. if it's ok in direct requests, then it's probably proxy indeed. try cleaning its cache
HaruCode
also hostnames might be the problem
HaruCode
you might see no error because you send a different hostname in your request
HaruCode
or it's your result that is cached and the actual state is bad
eyedeekay
Could be
eyedeekay
not the state, making my requests with wget, should be no cache
HaruCode
free space or config on proxy then
HaruCode
can't say for sure without looking. try tracing the traffic between the proxy and backend
eyedeekay
Proxy is trying to talk to the wrong hostname, but since it worked before and didn't change, and I can still reach it from the proxy host with both wget and curl, so still some kind of mystery to be solved
HaruCode
not quite get it: what hasn't changed and what can be reached? and if you didn't change something it doesn't mean other admin didn't. if there are other admins, of course
HaruCode
and the wrong hostname might me not related to this site. or it's a bad sign meaning the config has been tampered with. depends on whether you know this "wrong" hostname
eyedeekay
Proxy trying to reach the same hostname as it always has, but it isn't able to. But, if I log in to the proxy host and try the same hostname with curl, it works just fine
eyedeekay
Looking for clues in the config files
HaruCode
aha. local dns issues are likely
HaruCode
try using tcpdump to compare how the proxy and curl resolve the name
HaruCode
and maybe grepping the configs for the resulting ip
HaruCode
and if the name resolution is correct, dump the traffic to see differences in http streams
eyedeekay
OK got the name resolution fixed, according to the logs all requests for su3 files have been successful for about 10 minutes and no more failures to connect to the news host, but it's still 503'ing on the homepage
dr|z3d
if it's nginx, check /var/log/nginx/error.log or wherever it's stored, might shed some light.
eyedeekay
It's apache2 unfortunately, I'm much better at nginx but I'll get it
HaruCode
http-error.log should help
eyedeekay
I'm looking around in the logs for clues too
HaruCode
if it's apache, necessary modules might have not been loaded
HaruCode
of the dependencies for the modlue used, like php or perl additional modules
eyedeekay
Only new information I've got so far is a warning "Flask-Cache: CACHE_TYPE is set to null"
HaruCode
that only means the internal cache is disabled
eyedeekay
Yeah it seemed unrelated when I stuck it in a search engine
HaruCode
python modules and updates can be tricky. maybe an update happened that broke it
HaruCode
the http(d)-error.log has no details?
HaruCode
usually the problem is unintended python version switch
eyedeekay
Nothing new in the httpd error logs unfortunately
HaruCode
well, reinstalling the related packages or rebuilding from sources might reveal the problem, but I can't say how exactly it should be done, since it's os-dependent
eyedeekay
Yeah I'm always breaking python stuff that way, gotta be super-careful
HaruCode
check which python versions you have and which is the default. the wrong python is number 1 in the list of error caues
HaruCode
*causes
eyedeekay
about to be sick, back in ~20
eyedeekay
That sucked
StormyCloud
eyedeekay: You all good?
eyedeekay
Yeah I'll be fine
eyedeekay
Hey that's a little more helpful, 429 denied, access-control-allow-origin
eyedeekay
Got it to show me something useful, pretty sure if I fix that we're back to normal...
eyedeekay
Nope, I was wrong, fixed the 503's but the 429's are still there
eyedeekay
Nope, wrong about that too
eyedeekay
Damn
eyedeekay
and yet, every single su3 download seems to work without incident
eyedeekay
It is down to just the script that generates the stats svgs