We contracted with an outside firm that does a lot of SASIxp customization to write the app. It went into pilot at one elementary last year and did very well. We started a limited rollout over the winter at a couple of the high schools and ran into a performance problem that generated a rewrite of how the app queries SASI for data (it now batches all the students in over night and only sends record changes after the teacher has completed taking attendance for all students).
Tuesday morning we went live with HIPSTA across the entire school district. At around eight forty five Tuesday HIPSTA went *KABOOM* Total freeze-up. The problem cleared by around eleven AM, and then came right back again Wednesday morning. I got dragged into the troubleshooting along with several other folks. The phones were either just sitting there saying "Requesting page" or they were getting HTTP timeouts. It sounded like a web server resource problem, so that's how I started looking at it.
Over lunch I remembered that Microsoft's Internet Information Server (IIS) had some tuning parameters, and the default was not "run like you're a huge web server". So, we had a look at the IIS config when we got back from McComfort food. This, essentially, is it:
Not much there, really. And the descriptions are so helpful. The help for this control is only a smidge more enlightening. Having no other real option, we flicked it from the middle (default) setting to the right (high) setting. There are other things you can tweak if you want to go registry diving. The metric "hits per day" is particularly unhelpful. Events don't pile up over the course of twenty four hours to all be handled at once. They happen second by second and are responded to second by second. The break point of 100,000 hits per day comes down to 1.57 hits per second. Thursday morning's logs showed periods of sustained three to five hits per second activity. That, and positive feedback from the SASI admin, demonstrated that it was the right adjustment.
I'd love to claim victory, but we're still not out of the woods. HIPSTA ran ok on Thursday and Friday mornings, with sporadic calls coming from teachers unable to get into the app. The advice of "try it again in five minutes" seemed to be working. The app designer thinks that he may be hitting a wall with HIPSTA's internal Access database (a max of 15 concurrent accesses to the database). We'll probably be moving it over to a MS SQL server (which is where it should have been from the start, IMO), and limping along as we are until then.
I won't waste your time by whining about how Apache is better than IIS, but I will say that this is another fine example of a Microsoft product that is miserably opaque and damned hard to debug and tune. For treating us IT folk like morons who must be shielded from all those complex "technology things" I say thanks Bill.
