Go back to previous page
Forum URL: http://www.eyrie-productions.com/Forum/dcboard.cgi
Forum Name: The Forum Itself
Topic ID: 84
#0, Forum back in service... kind of.
Posted by Gryphon on Apr-21-14 at 12:03 PM
LAST EDITED ON Apr-21-14 AT 03:44 PM (EDT)
 
So!

Here's the short version: The Forum still appears to be somewhat broken, but though we have a few leads, we're not sure what's actually going wrong with it yet, nor whether it's an artifact of the great migration, the lesser migration (the latter seems unlikely based on certain matters of timing), or what. This is still under investigation, but for the moment, what we're going to do is turn it back on and observe its behavior under operating conditions for a while.

For the moment, the UF Mini-Stories forum (where the worst of the damage was happening) is still shut off while Dave investigates. We restored it from a recent backup last night and it was still slightly broken, at which point I ran the Forum's automated "rebuild database" function and it went completely to bits in exactly the same way as it had the first time, which was both alarming and informative. Dave then restored it from backups again and is now digging around in the backend.

Pardon our dust, etc.

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


#1, RE: Forum back in service... kind of.
Posted by Offsides on Apr-21-14 at 02:28 PM
In response to message #0
Sounds like your database got corrupted somehow, possibly due to version issues from the big migration? In any event, good luck, fixing a database can be a royal pain.

Offsides

[...] in order to be a realist you must believe in miracles.
-- David Ben Gurion
EPU RCW #π
#include <stdsig.h>


#2, RE: Forum back in service... kind of.
Posted by Gryphon on Apr-21-14 at 03:58 PM
In response to message #1
>Sounds like your database got corrupted somehow, possibly due to
>version issues from the big migration?

Mmmmaybe. The thing is, as I've said before, the Forum doesn't use a database database; it's really just a bunch of text files with sekrit markup in them that gets parsed by some scripts. Based on its behavior during our tests last night, I'm starting to suspect the opposite is actually true - that something happened to a bunch of the text files, munging the markup in them that the forum scripts use to build the "database" in the first place. That would explain why running a database rebuild made the problem worse, I think.

Like I say, work in progress. Today is the first day of the spring semester's penultimate week of classes, so I have papers to finish and final exams to prepare for, and naturally my creative lobe has taken this opportunity to find the traction that eluded it through most of the "slow" part of the semester on various non-academic projects.* Dave, who's doing most of the heavy lifting, is only able to work on it in his spare time. Neither one of us really knows how this crufty old software works, which, as many of you may know, is something of an obstacle to troubleshooting, particularly timely and effective troubleshooting.

Some of you have offered your help, and I appreciate it, but as yet I haven't thought of a way of managing the access required to make that work; another thing I don't have optimal mental bandwidth for dealing with due to the above. (Besides which, it's not really my call anyway; it's Dave's server.) So for right now, I think we'll have to hold off on that.

Ultimately, the real solution to the problem may be to move to a more modern forum platform. There are a lot of reasons why I'm not very comfortable with that, but on the other hand, randomly losing content is clearly not something that can be allowed to happen. Losing the What's Past Is Prologue reaction thread was painful, but losing actual stories is just not on.

I suppose it could be argued that it's my own fault for using a discussion forum as a content-providing tool, but dammit, it's handy.

--G.
* Always waste time when you haven't got any. Rule 408: Time is not the boss of you.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


#3, RE: Forum back in service... kind of.
Posted by zwol on Apr-21-14 at 07:35 PM
In response to message #0
A thought to pass along to Dave, perhaps: The age of the software, combined with the location where the problem seems to be worst (Mini-Stories) makes me wonder if you're looking at a bug in DCForum's handling of threads containing non-ASCII characters.

#4, RE: Forum back in service... kind of.
Posted by Gryphon on Apr-21-14 at 07:48 PM
In response to message #3
>A thought to pass along to Dave, perhaps: The age of the software,
>combined with the location where the problem seems to be worst
>(Mini-Stories) makes me wonder if you're looking at a bug in DCForum's
>handling of threads containing non-ASCII characters.

Hmm. I suppose that's possible, but one does wonder why it would be doing it now. We've been using ampersand entities since very early on, and not just in the Mini-Stories, and DCF may be old by software standards, but it's not older than ISO-Latin-1. :)

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


#5, RE: Forum back in service... kind of.
Posted by zwol on Apr-21-14 at 08:02 PM
In response to message #4
LAST EDITED ON Apr-21-14 AT 08:02 PM (EDT)
 
Ampersand entities are ASCII in the storage; they just get converted to accented letters or Greek or whatever by the browser. Are the more recent threads perhaps using the raw characters instead?

(It is equally possible that I am seeing the ghosts of such bugs that I and/or my SO have had to fix in other software. It is depressingly easy to screw up non-ASCII handling even in quite recent code.)


#6, RE: Forum back in service... kind of.
Posted by jonathanlennox on Apr-21-14 at 10:07 PM
In response to message #5
>Ampersand entities are ASCII in the storage; they just get
>converted to accented letters or Greek or whatever by the browser.
>Are the more recent threads perhaps using the raw characters instead?
>
>(It is equally possible that I am seeing the ghosts of such bugs that
>I and/or my SO have had to fix in other software. It is depressingly
>easy to screw up non-ASCII handling even in quite recent code.)

Hm, yes, that seems plausible... if you've got one API that's counting UTF-8 characters -- and another one that's counting bytes -- and the code is using these counts to index into a flat text file, and expecting them to agree...that seems very much like something that could go sideways quite easily.

It also seems like the sort of thing where a system upgrade could "helpfully" change APIs' behaviors in the scripting language.


#7, RE: Forum back in service... kind of.
Posted by jonathanlennox on Apr-21-14 at 10:32 PM
In response to message #6
>Hm, yes, that seems plausible... if you've got one API that's counting
>UTF-8 characters -- and another one that's counting bytes -- and the
>code is using these counts to index into a flat text file, and
>expecting them to agree...that seems very much like something that
>could go sideways quite easily.
>
>It also seems like the sort of thing where a system upgrade could
>"helpfully" change APIs' behaviors in the scripting language.

Further investigation, inferring based on some Googling that the forum is written in Perl -- Perl has indeed mucked about a lot with its UTF-8 support in recent-ish versions (recent compared to the age of the forum software, anyway).

Does the forum's environment have any of PERL_UNICODE, LANG, or any LC_ variables set?

Alternatively, if this undirected speculation -- debugging without information -- is unhelpful and/or annoying, please tell me to stop!


#8, RE: Forum back in service... kind of.
Posted by Senji on Apr-22-14 at 05:17 AM
In response to message #7
>Further investigation, inferring based on some Googling that the forum
>is written in Perl -- Perl has indeed mucked about a lot with its
>UTF-8 support in recent-ish versions (recent compared to the age of
>the forum software, anyway).

Perl's Unicode support is, by now, almost fractally wacky (and also rather insane). I agree that it might be a fertile place for bugs to appear.

S.


#9, RE: Forum back in service... kind of.
Posted by JeanneHedge on Apr-22-14 at 11:41 AM
In response to message #0
I think it's a good thing that you were putting "stay away from the forum" notices on the What's New page. Have to admit, I was another who missed that the Announcement icon was going off (or it didn't register) until I had been poking in the forum for a couple minutes.

Until there's an all-clear, I may just continue to check there before coming here.


Jeanne


Jeanne Hedge
http://www.jhedge.com
"Never give up, never surrender!"


#10, RE: Forum back in service... kind of.
Posted by Berrik on Apr-22-14 at 08:46 PM
In response to message #9
Why don't you just archive the old forums and install some newer, better forum software? I can't imagine storing literally everything in text files is terribly secure, especially since the company that makes the forum software is out of business.

#11, RE: Forum back in service... kind of.
Posted by Senji on Apr-23-14 at 05:57 AM
In response to message #10
>Why don't you just archive the old forums and install some newer,
>better forum software? I can't imagine storing literally everything in
>text files is terribly secure, especially since the company that makes
>the forum software is out of business.

There is no better forum software, only newer :-). All forum software is shite, you just get to pick which kind of shiteness you want.

Also I'd much rather have flat text files than structured binary crap; in the worst case it gives you migration options.

S.


#12, RE: Forum back in service... kind of.
Posted by laudre on Apr-23-14 at 06:46 AM
In response to message #11
> Also I'd much rather have flat text files than structured binary
>crap; in the worst case it gives you migration options.

Sadly, near as I can tell, support for flat text files for forum data storage is a thing of the past. Wikipedia's list only shows two out of a fairly long list that support flat text, and both of those (Ikonboard and UBB.classic) are approximately as dead as DCForum. Everything that's still in active maintenance seems to run on SQL (usually MySQL).


"Mathematics brought rigor to economics. Unfortunately, it also brought mortis."
- Kenneth Boulding


#13, RE: Forum back in service... kind of.
Posted by Gryphon on Apr-23-14 at 12:07 PM
In response to message #0
OK, got a couple of minutes between classes, so I figured I'd update you on the latest. As you may have seen in the announcement, Dave has restored the damaged threads from the latest good backups he could find, and apart from some view-count losses, there hasn't been any particular damage due to the time warpage involved (we only had to reset the threads that were affected, not the whole Forum).

We're still not sure what caused the damage, though there are theories. The threads involved seem to have been affected at random, and the clustering in ForumID24, though pronounced, was not as exclusive as it looked in the heat of the moment, so there's every reason to suspect at this point that it was in fact not the Forum losing its mind, but in fact the Forum reacting normally to damaged source files. The question of how those files were damaged remains open, but based on Dave's findings amid the time stamps and whatnot, he's hopeful that it was a single incident connected with last week's migration to an SSD-based hosting platform.

So, we're going to keep an eye on it (and we have a more aggressive backup strategy now), but at the moment (touch wood), it appears that normal operations may resume.

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


#15, RE: Forum back in service... kind of.
Posted by JeanneHedge on Apr-23-14 at 06:09 PM
In response to message #13
Was What's Past... recoverable?

Jeanne


Jeanne Hedge
http://www.jhedge.com
"Never give up, never surrender!"


#16, RE: Forum back in service... kind of.
Posted by Gryphon on Apr-23-14 at 06:15 PM
In response to message #15
>Was What's Past... recoverable?

Alas, no. That happened last December; we only have backups since the Great Migration in February, so I fear the Prologue reactions are gone forever. Everything that was affected in the current weirdness has been repaired, though.

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


#17, RE: Forum back in service... kind of.
Posted by JeanneHedge on Apr-25-14 at 03:01 PM
In response to message #16
While this was going on, I was poking around on archive.org, to see what and wasn't there. There seemed to be a big gap of board material they didn't have, and then they had a few of the more recent things. Is there some sort of anti-bot coding here (that didn't used to be here) that's blocking them? (IIRC, Google cache had a similar thing going on)

Of course, I'm guessing any anti-bot codes is actually anti-spammer codes, and archive.org may have been caught in the crossfire.


Jeanne


Jeanne Hedge
http://www.jhedge.com
"Never give up, never surrender!"


#14, RE: Forum back in service... kind of.
Posted by trboturtle2 on Apr-23-14 at 03:51 PM
In response to message #0
Shows you how clueless I am -- I didn't know there as a problem until i read this thread.....

Craig