HomePage RecentChanges

metamath website issues

This page can be used to discuss technical problems with the Metamath website and mirrors.


us2.metamath.org problem

There is a problem with us2.metamath.org: it is unavailable half of the time. And this half corresponds to the day in Europe (and consequently in Africa too). – fl

On Sunday 15 Jun, there was a 4-hour power outage from 09:00 to 13:00 EDT in my neighborhood due to a bad transformer. I checked the logs, and there have been no other gaps in the hourly RSS feed updates from Israel (at 00:00, 01:00,… 23:00 +- 1 minute) for the last two weeks. So either there are Internet outages in-between those times I am unaware of, or there is some other problem between here and Europe/Africa. Perhaps you can report to me the exact time if you see an outage, and I will look at the logs to see if other activity stopped as well. – norm 18 Jun 2008

Now the page is unavailable (15:10 in Paris). – fl

The access.log showed activity from UK (but not from France) at 15:15 Paris time = 13:15 UTC = 09:15 EDT. Is the problem that us2.metamath.org didn't respond, or that the domain name lookup failed (two different problems)? Next time this happens, could you try http://207.172.220.155:8888 instead of http://us2.metamath.org:8888 and let me know what happens? Does anyone else reading this have a problem? – norm 18 Jun 2008

Both are unavailable (Paris 9h17 UTC). I don't know the source of the problem maybe a domain name lookup failure. When you click on the link nothing happens. – fl 19-Jun-2008

I have received another report of the problem from the Netherlands. It is not a DNS problem. The HTTP connection actually gets initialized, but then for unknown reason the TCP conversation stops. – norm 19 Jun 2008

This has only happened to me – about – once during the past year, and I check the website for updates once or twice each day.
.
Question: why do you use port 8888 for the US2 website? I am wondering (unlikely), perhaps, if you might try an experiment and create a clone of the homepage with port 80 and see if the problem occurs simultaneously on 8888 and 80. Other than that, I wonder if your Debian is out of date (I am sure you have covered that situation though – mostly I am just curious about your diagnostic process. Thx.) --ocat 19-Jun-2008
I use port 8888 because my ISP blocks incoming port 80 for anyone who doesn't pay for a high-priced "commercial" Internet connection. As for Debian being outdated, it has worked for years without a problem, but it will eventually be something to look at.

Today I replaced the router with another brand. Let me know if this affects the problem. – norm 19 Jun 2008

No it still doesn't work. Do you have some restrictions in your parameters that wouldn't allow people to connect at certain hours of the day ? Firewalls for instance do that sort of things ? Or your ISP has a new policy ? – fl 20-Jun-2008
There aren't any restrictions, and the firewall does not selectively block anyone either by time of day or by IP address. As for a new ISP policy, it would be odd to filter only connections from Europe - unless one is a conspiracy theorist - although they won't help me with a server problem since I don't have a commercial account. – norm
Definition "conspiracy": "An agreement to perform together an illegal, wrongful, or subversive act." I do not believe these are very rare. Quite the opposite. So theorizing about them ought to be as natural as breathing, at least for Thinkers. But in this case an unintentional malfunction is the likely cause. Still… perhaps the conspirators' servers overload at certain times of the day :-) --ocat.

For the time being, the http://gr.metamath.org mirror will be updated daily starting at 19:00 EST (23:00 UTC, 01:00 Paris) and usually ending before 19:30. – norm 21 Jun 2008

In a "mysterious" situation the key to resolution is gathering more evidence, which may mean conducting experiments. Inside the US my access to US2 has been solid, with only one "unavailable" situation in a year (fuzzy recollection on that.) Logically, there are three locations of the problem: user computer, server computer, and everything in the middle. Since the two reported instances of this situation in the current timeframe are European, the most suspcious culprit is "everything in the middle". I would be curious to know if us.metamath.org (US based) is unavailable at the same time that us2.metamath.org is unavailable. I wonder if the users reporting the problem use PC firewalls and can examine then report the firewall log data for problem analysis. It may be that there is some sort of filtering going on between the US and Europe which chokes sometimes on the US2..:8888 communications – that is, the filtering is different than when US..:80 is used. There is also a possibility of "0wn3rship" of someone's computer somewhere, with attempts to do a man-in-the-middle attack – so I wonder if the European users who are reporting the problem could attempt using a proxy – perhaps use TOR – to access US2 when a problem is encountered using the normal methods. --ocat 21-Jun-2008
The us.metamath.org volunteer could no longer offer the mirror, so now I map it to either au.metamath.org or gr.metamath.org, whichever seems to be working the most reliably. So in a sense us.metamath.org is the "most reliable" mirror, but is physically located in either Australia or Greece. As for us2, there is a Netherlands volunteer who has been able to reproduce the problem from his end and will be doing some troubleshooting over the next few days. – norm

I added ports 88 and 443, so the server now recognizes http://us2.metamath.org:8888 and http://us2.metamath.org:88 and http://us2.metamath.org:443 identically. This may tell us if the problem is related to the port number. If anyone sees them behave differently, let me know. – norm 21 Jun 2008

Here are two more experiments testing the hypothesis that some ISP's are interfering with suspected transborder file-sharing activities – anything other than port 80 accesses.
It would be strange if they were selectively blocking only at certain times of the day, since the behavior of the problem seems to depend on that.
1) As you know, the main vanilla TCP ports are 80 (http), 443 (https) and 53 (DNS). I don't expect 88 to fare better than 8888 – both are non-80, hence non- standard. Does your ISP prevent 443 sessions? Can you provide an HTTPS session for US2? It would be interesting to see if HTTPS is blocking.
Interestingly, it does seem that my ISP doesn't block 443. So I added http://us2.metamath.org:443 (which is not SSL though - one thing at a time…).
2) Provide temporary readonly login access to the US2 server on some non-critial directory, such as /ocat/mmj2. Give to fl and your Netherlands volunteer. If they can access login at us2.metamath.org but still cannot access the webpages via a browser then that would be very interesting.
This has already been done behind the scenes. The NL person can get in via ssh and is doing various tcpdump, nmap, etc. things to see what's happening.

A quick update on what I found out in the last couple of days:

     23:17:36.580041 IP 10.31.104.10.60171 > 207-172-220-155.c3-0.lex-ubr2.sbo-lex.ma.cable.rcn.com.8888: S 2048871615:2048871615(0) win 5840 <mss 1460,sackOK,timestamp 3354877477 0,nop,wscale 2>
     23:17:36.580204 IP 207-172-220-155.c3-0.lex-ubr2.sbo-lex.ma.cable.rcn.com.8888 > 10.31.104.10.60171: R 0:0(0) ack 2048871616 win 5840 <mss 1460,sackOK,timestamp 3354877477 0,nop,wscale 2>

That's all I can find out for now, unfortunately I have no more leads to go on. We still have a mystery. I'm thinking that Norm's ISP is doing the blocking, because several routers exhibit the same problem, and the us2 box itself does not even see the 8888 packets. It could be further upstream from Norm, but that seems less likely: then other customers might also have complained about blocked ports.

So the next stop in this investigation seems to be Norm's ISP.

If anyone has more information, we'd appreciate it.

--marnix

More Evidence Gathering

In the US, Comcast has admitted interfering with P2P by injecting packets that – I don't know the specific details – terminate or reset the "conversation". They claim not to "block" and that a user can eventually complete the download with perseverance. So, in the Comcast situation the problem is – or is claimed to be – intermittent. It looks like Norm's ISP is a cable provider, and we already know that they block incoming port 80 because metamath.org is not paid for with a "commercial" fee.

They are upfront about the fact that port 80 is blocked. It would surprise me if they are purposely and surreptitiously blocking another port, especially only from certain countries, although I suppose anything is possible. My guess is that some router in their farm is misconfigured or defective. The only thing I can think of is for people with the problem to do a "traceroute us2.metamath.org" ("tracert us2.metamath.org" on Windows from the Command Prompt) to see if there is a particular router that is always there when the problem occurs and never there when the problem doesn't occur.

So, the question is, what types of internet connections are being used with us2.metamath.org, both successful and unsuccessful? For example, my internet connection is dial-up, and I almost never have any problems accessing us2.metamath.org. Are the problems occurring only on high-speed internet connections, and what are those speeds?

I generated a web report for us2 and put it here: http://us2.metamath.org:88/1.html. Make of it what you will. Out of these, there have only been 2 reports of this problem, one from NL and one from FR, although of course casual users aren't going to report a problem. (You may be interested that there have been 297 mmj2.zip downloads since it was added around Oct.)

I am also curious about what is the total bandwidth usage of the internet connection which us2.metamath.org consumes, including both the Metamath and any other activities – such as foldit@home or whatever.

And is it possible that something like foldit@home is active during the same timeframe in which problems are being reported?

At the time Marnix verified the problem from his end, I don't think there was any other activity at all going on.

Also, how many unique IP addresses use us2.metamath.org on a regular basis, not counting the occasional spikes due to slashdot, etc. It would be interesting to know what percentage of regular users are experiencing this intermittent problem (we know that most users never report internet problems.)

--ocat 25-Jun-2008

I will keep ports 88, 443, and 8888 open indefinitely as a workaround. I made port 88 the "default" since no one has reported a problem with it yet. – norm 26 Jun 2008