[ EPU Foyer ] [ Lab and Grill ] [ Bonus Theater!! ] [ Rhetorical Questions ] [ CSRANTronix ] [ GNDN ] [ Subterranean Vault ] [ Discussion Forum ]

Eyrie Productions, Unlimited

Subject: "Forum back in service... kind of."     Previous Topic | Next Topic
Printer-friendly copy    
Conferences The Forum Itself Topic #84
Reading Topic #84
Gryphonadmin
Charter Member
13752 posts
Apr-21-14, 12:03 PM (EDT)
Click to EMail Gryphon Click to send private message to Gryphon Click to view user profileClick to add this user to your buddy list  
"Forum back in service... kind of."
 
   LAST EDITED ON Apr-21-14 AT 03:44 PM (EDT)
 
So!

Here's the short version: The Forum still appears to be somewhat broken, but though we have a few leads, we're not sure what's actually going wrong with it yet, nor whether it's an artifact of the great migration, the lesser migration (the latter seems unlikely based on certain matters of timing), or what. This is still under investigation, but for the moment, what we're going to do is turn it back on and observe its behavior under operating conditions for a while.

For the moment, the UF Mini-Stories forum (where the worst of the damage was happening) is still shut off while Dave investigates. We restored it from a recent backup last night and it was still slightly broken, at which point I ran the Forum's automated "rebuild database" function and it went completely to bits in exactly the same way as it had the first time, which was both alarming and informative. Dave then restored it from backups again and is now digging around in the backend.

Pardon our dust, etc.

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top

  Subject     Author     Message Date     ID  
  RE: Forum back in service... kind of. Offsides Apr-21-14 1
     RE: Forum back in service... kind of. Gryphonadmin Apr-21-14 2
  RE: Forum back in service... kind of. zwol Apr-21-14 3
     RE: Forum back in service... kind of. Gryphonadmin Apr-21-14 4
         RE: Forum back in service... kind of. zwol Apr-21-14 5
             RE: Forum back in service... kind of. jonathanlennox Apr-21-14 6
                 RE: Forum back in service... kind of. jonathanlennox Apr-21-14 7
                     RE: Forum back in service... kind of. Senji Apr-22-14 8
  RE: Forum back in service... kind of. JeanneHedge Apr-22-14 9
     RE: Forum back in service... kind of. Berrik Apr-22-14 10
         RE: Forum back in service... kind of. Senji Apr-23-14 11
             RE: Forum back in service... kind of. laudre Apr-23-14 12
  RE: Forum back in service... kind of. Gryphonadmin Apr-23-14 13
     RE: Forum back in service... kind of. JeanneHedge Apr-23-14 15
         RE: Forum back in service... kind of. Gryphonadmin Apr-23-14 16
             RE: Forum back in service... kind of. JeanneHedge Apr-25-14 17
  RE: Forum back in service... kind of. trboturtle2 Apr-23-14 14

Conferences | Topics | Previous Topic | Next Topic
Offsides
Charter Member
1148 posts
Apr-21-14, 02:28 PM (EDT)
Click to EMail Offsides Click to send private message to Offsides Click to view user profileClick to add this user to your buddy list  
1. "RE: Forum back in service... kind of."
In response to message #0
 
   Sounds like your database got corrupted somehow, possibly due to version issues from the big migration? In any event, good luck, fixing a database can be a royal pain.

Offsides

[...] in order to be a realist you must believe in miracles.
-- David Ben Gurion
EPU RCW #π
#include <stdsig.h>


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
Gryphonadmin
Charter Member
13752 posts
Apr-21-14, 03:58 PM (EDT)
Click to EMail Gryphon Click to send private message to Gryphon Click to view user profileClick to add this user to your buddy list  
2. "RE: Forum back in service... kind of."
In response to message #1
 
   >Sounds like your database got corrupted somehow, possibly due to
>version issues from the big migration?

Mmmmaybe. The thing is, as I've said before, the Forum doesn't use a database database; it's really just a bunch of text files with sekrit markup in them that gets parsed by some scripts. Based on its behavior during our tests last night, I'm starting to suspect the opposite is actually true - that something happened to a bunch of the text files, munging the markup in them that the forum scripts use to build the "database" in the first place. That would explain why running a database rebuild made the problem worse, I think.

Like I say, work in progress. Today is the first day of the spring semester's penultimate week of classes, so I have papers to finish and final exams to prepare for, and naturally my creative lobe has taken this opportunity to find the traction that eluded it through most of the "slow" part of the semester on various non-academic projects.* Dave, who's doing most of the heavy lifting, is only able to work on it in his spare time. Neither one of us really knows how this crufty old software works, which, as many of you may know, is something of an obstacle to troubleshooting, particularly timely and effective troubleshooting.

Some of you have offered your help, and I appreciate it, but as yet I haven't thought of a way of managing the access required to make that work; another thing I don't have optimal mental bandwidth for dealing with due to the above. (Besides which, it's not really my call anyway; it's Dave's server.) So for right now, I think we'll have to hold off on that.

Ultimately, the real solution to the problem may be to move to a more modern forum platform. There are a lot of reasons why I'm not very comfortable with that, but on the other hand, randomly losing content is clearly not something that can be allowed to happen. Losing the What's Past Is Prologue reaction thread was painful, but losing actual stories is just not on.

I suppose it could be argued that it's my own fault for using a discussion forum as a content-providing tool, but dammit, it's handy.

--G.
* Always waste time when you haven't got any. Rule 408: Time is not the boss of you.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
zwol
Member since Feb-24-12
62 posts
Apr-21-14, 07:35 PM (EDT)
Click to EMail zwol Click to send private message to zwol Click to view user profileClick to add this user to your buddy list  
3. "RE: Forum back in service... kind of."
In response to message #0
 
   A thought to pass along to Dave, perhaps: The age of the software, combined with the location where the problem seems to be worst (Mini-Stories) makes me wonder if you're looking at a bug in DCForum's handling of threads containing non-ASCII characters.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
Gryphonadmin
Charter Member
13752 posts
Apr-21-14, 07:48 PM (EDT)
Click to EMail Gryphon Click to send private message to Gryphon Click to view user profileClick to add this user to your buddy list  
4. "RE: Forum back in service... kind of."
In response to message #3
 
   >A thought to pass along to Dave, perhaps: The age of the software,
>combined with the location where the problem seems to be worst
>(Mini-Stories) makes me wonder if you're looking at a bug in DCForum's
>handling of threads containing non-ASCII characters.

Hmm. I suppose that's possible, but one does wonder why it would be doing it now. We've been using ampersand entities since very early on, and not just in the Mini-Stories, and DCF may be old by software standards, but it's not older than ISO-Latin-1. :)

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
zwol
Member since Feb-24-12
62 posts
Apr-21-14, 08:02 PM (EDT)
Click to EMail zwol Click to send private message to zwol Click to view user profileClick to add this user to your buddy list  
5. "RE: Forum back in service... kind of."
In response to message #4
 
   LAST EDITED ON Apr-21-14 AT 08:02 PM (EDT)
 
Ampersand entities are ASCII in the storage; they just get converted to accented letters or Greek or whatever by the browser. Are the more recent threads perhaps using the raw characters instead?

(It is equally possible that I am seeing the ghosts of such bugs that I and/or my SO have had to fix in other software. It is depressingly easy to screw up non-ASCII handling even in quite recent code.)


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
jonathanlennox
Charter Member
170 posts
Apr-21-14, 10:07 PM (EDT)
Click to EMail jonathanlennox Click to send private message to jonathanlennox Click to view user profileClick to add this user to your buddy list  
6. "RE: Forum back in service... kind of."
In response to message #5
 
   >Ampersand entities are ASCII in the storage; they just get
>converted to accented letters or Greek or whatever by the browser.
>Are the more recent threads perhaps using the raw characters instead?
>
>(It is equally possible that I am seeing the ghosts of such bugs that
>I and/or my SO have had to fix in other software. It is depressingly
>easy to screw up non-ASCII handling even in quite recent code.)

Hm, yes, that seems plausible... if you've got one API that's counting UTF-8 characters -- and another one that's counting bytes -- and the code is using these counts to index into a flat text file, and expecting them to agree...that seems very much like something that could go sideways quite easily.

It also seems like the sort of thing where a system upgrade could "helpfully" change APIs' behaviors in the scripting language.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
jonathanlennox
Charter Member
170 posts
Apr-21-14, 10:32 PM (EDT)
Click to EMail jonathanlennox Click to send private message to jonathanlennox Click to view user profileClick to add this user to your buddy list  
7. "RE: Forum back in service... kind of."
In response to message #6
 
   >Hm, yes, that seems plausible... if you've got one API that's counting
>UTF-8 characters -- and another one that's counting bytes -- and the
>code is using these counts to index into a flat text file, and
>expecting them to agree...that seems very much like something that
>could go sideways quite easily.
>
>It also seems like the sort of thing where a system upgrade could
>"helpfully" change APIs' behaviors in the scripting language.

Further investigation, inferring based on some Googling that the forum is written in Perl -- Perl has indeed mucked about a lot with its UTF-8 support in recent-ish versions (recent compared to the age of the forum software, anyway).

Does the forum's environment have any of PERL_UNICODE, LANG, or any LC_ variables set?

Alternatively, if this undirected speculation -- debugging without information -- is unhelpful and/or annoying, please tell me to stop!


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
Senji
Member since Apr-27-07
155 posts
Apr-22-14, 05:17 AM (EDT)
Click to EMail Senji Click to send private message to Senji Click to view user profileClick to add this user to your buddy list  
8. "RE: Forum back in service... kind of."
In response to message #7
 
   >Further investigation, inferring based on some Googling that the forum
>is written in Perl -- Perl has indeed mucked about a lot with its
>UTF-8 support in recent-ish versions (recent compared to the age of
>the forum software, anyway).

Perl's Unicode support is, by now, almost fractally wacky (and also rather insane). I agree that it might be a fertile place for bugs to appear.

S.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
JeanneHedge
Charter Member
828 posts
Apr-22-14, 11:41 AM (EDT)
Click to EMail JeanneHedge Click to send private message to JeanneHedge Click to view user profileClick to add this user to your buddy list  
9. "RE: Forum back in service... kind of."
In response to message #0
 
   I think it's a good thing that you were putting "stay away from the forum" notices on the What's New page. Have to admit, I was another who missed that the Announcement icon was going off (or it didn't register) until I had been poking in the forum for a couple minutes.

Until there's an all-clear, I may just continue to check there before coming here.


Jeanne


Jeanne Hedge
http://www.jhedge.com
"Never give up, never surrender!"


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
Berrik
Member since Jul-10-07
54 posts
Apr-22-14, 08:46 PM (EDT)
Click to EMail Berrik Click to send private message to Berrik Click to view user profileClick to add this user to your buddy list Click to send message via AOL IM  
10. "RE: Forum back in service... kind of."
In response to message #9
 
   Why don't you just archive the old forums and install some newer, better forum software? I can't imagine storing literally everything in text files is terribly secure, especially since the company that makes the forum software is out of business.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
Senji
Member since Apr-27-07
155 posts
Apr-23-14, 05:57 AM (EDT)
Click to EMail Senji Click to send private message to Senji Click to view user profileClick to add this user to your buddy list  
11. "RE: Forum back in service... kind of."
In response to message #10
 
   >Why don't you just archive the old forums and install some newer,
>better forum software? I can't imagine storing literally everything in
>text files is terribly secure, especially since the company that makes
>the forum software is out of business.

There is no better forum software, only newer :-). All forum software is shite, you just get to pick which kind of shiteness you want.

Also I'd much rather have flat text files than structured binary crap; in the worst case it gives you migration options.

S.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
laudre
Member since Nov-14-06
332 posts
Apr-23-14, 06:46 AM (EDT)
Click to EMail laudre Click to send private message to laudre Click to view user profileClick to add this user to your buddy list Click to send message via AOL IM  
12. "RE: Forum back in service... kind of."
In response to message #11
 
   > Also I'd much rather have flat text files than structured binary
>crap; in the worst case it gives you migration options.

Sadly, near as I can tell, support for flat text files for forum data storage is a thing of the past. Wikipedia's list only shows two out of a fairly long list that support flat text, and both of those (Ikonboard and UBB.classic) are approximately as dead as DCForum. Everything that's still in active maintenance seems to run on SQL (usually MySQL).


"Mathematics brought rigor to economics. Unfortunately, it also brought mortis."
- Kenneth Boulding


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
Gryphonadmin
Charter Member
13752 posts
Apr-23-14, 12:07 PM (EDT)
Click to EMail Gryphon Click to send private message to Gryphon Click to view user profileClick to add this user to your buddy list  
13. "RE: Forum back in service... kind of."
In response to message #0
 
   OK, got a couple of minutes between classes, so I figured I'd update you on the latest. As you may have seen in the announcement, Dave has restored the damaged threads from the latest good backups he could find, and apart from some view-count losses, there hasn't been any particular damage due to the time warpage involved (we only had to reset the threads that were affected, not the whole Forum).

We're still not sure what caused the damage, though there are theories. The threads involved seem to have been affected at random, and the clustering in ForumID24, though pronounced, was not as exclusive as it looked in the heat of the moment, so there's every reason to suspect at this point that it was in fact not the Forum losing its mind, but in fact the Forum reacting normally to damaged source files. The question of how those files were damaged remains open, but based on Dave's findings amid the time stamps and whatnot, he's hopeful that it was a single incident connected with last week's migration to an SSD-based hosting platform.

So, we're going to keep an eye on it (and we have a more aggressive backup strategy now), but at the moment (touch wood), it appears that normal operations may resume.

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
JeanneHedge
Charter Member
828 posts
Apr-23-14, 06:09 PM (EDT)
Click to EMail JeanneHedge Click to send private message to JeanneHedge Click to view user profileClick to add this user to your buddy list  
15. "RE: Forum back in service... kind of."
In response to message #13
 
   Was What's Past... recoverable?

Jeanne


Jeanne Hedge
http://www.jhedge.com
"Never give up, never surrender!"


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
Gryphonadmin
Charter Member
13752 posts
Apr-23-14, 06:15 PM (EDT)
Click to EMail Gryphon Click to send private message to Gryphon Click to view user profileClick to add this user to your buddy list  
16. "RE: Forum back in service... kind of."
In response to message #15
 
   >Was What's Past... recoverable?

Alas, no. That happened last December; we only have backups since the Great Migration in February, so I fear the Prologue reactions are gone forever. Everything that was affected in the current weirdness has been repaired, though.

--G.
-><-
Benjamin D. Hutchins, Co-Founder, Editor-in-Chief, & Forum Mod
Eyrie Productions, Unlimited http://www.eyrie-productions.com/
zgryphon at that email service Google has
Ceterum censeo Carthaginem esse delendam.


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
JeanneHedge
Charter Member
828 posts
Apr-25-14, 03:01 PM (EDT)
Click to EMail JeanneHedge Click to send private message to JeanneHedge Click to view user profileClick to add this user to your buddy list  
17. "RE: Forum back in service... kind of."
In response to message #16
 
   While this was going on, I was poking around on archive.org, to see what and wasn't there. There seemed to be a big gap of board material they didn't have, and then they had a few of the more recent things. Is there some sort of anti-bot coding here (that didn't used to be here) that's blocking them? (IIRC, Google cache had a similar thing going on)

Of course, I'm guessing any anti-bot codes is actually anti-spammer codes, and archive.org may have been caught in the crossfire.


Jeanne


Jeanne Hedge
http://www.jhedge.com
"Never give up, never surrender!"


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top
trboturtle2
Member since Jul-4-09
103 posts
Apr-23-14, 03:51 PM (EDT)
Click to EMail trboturtle2 Click to send private message to trboturtle2 Click to view user profileClick to add this user to your buddy list Click to send message via AOL IM  
14. "RE: Forum back in service... kind of."
In response to message #0
 
   Shows you how clueless I am -- I didn't know there as a problem until i read this thread.....

Craig

-----------------------------
Writer for BattleCorps.com/Fanfiction writer. AND all around semi-nice guy! Really!!


  Alert | IP Printer-friendly page | Edit | Reply | Reply With Quote | Top

Conferences | Topics | Previous Topic | Next Topic

[ YUM ] [ BIG ] [ ??!? ] [ RANT ] [ GNDN ] [ STORE ] [ FORUM ] [ VAULT ]

version 3.3 © 2001
Eyrie Productions, Unlimited
Benjamin D. Hutchins
E P U (Colour)