Older History of Changes...

LAST DUMP: JUNE 18/96:

After a week on the NEW HONKIN' SERVER, we were getting FLOODED with hits by one user (250,000 per day). The amount of acitivity on the disk rendered the NFS-mounting useless (Geo-counts were on the same volume as the standard daemon's access_ log files). I tried to compensate by configuring an FDDI card on the new machine, but it matters worse --- it just got "no NFS volume available" messages faster.

I migrated all of the log files from the old server to the new one, which took time. Then, the volume wasn't big enough, so I devised a new partitioning scheme that could have either sped things up, or broke things for good.

The STATS pages will have to stay broken for a few days, but I think we're back in business. Email me if you have noticed any difference one way or the other.

Stopped using the FDDI. For some reason, it was KILLING the performance.

In doing so, I lost DNS control for the night, and your "last N Users" are only listed by IP address - not their names. Sorry 'bout that. Should be fixed now.

Fixed the nightly stats/fill program that checks references per day, URLs etc., so the stats should be more up-to-date.

I inadvertently removed my gif conversion libraries, while re-positioning some disk space (I'm running out). From Mar4-6pm to Mar5-9:30am, the counter was working, just not displaying properly. It should be fixed now. I wonder why I only got two email messages on this.

A little addition to the nightly routine should attempt to re-sync counters with a probability of 1 chance in 10. That means, on average, every 10 days, counters that have gotten out-of-sync between the daily reports and the normal entry will be set to the higher number.

Registered counters can use the "resync" button from housekeeping anytime they want.

Feb 14/95 DUMP:

Worked out a new "find routine". Added to http://www.ualberta.ca/htbin/Geo-stats. It has a few bugs in it. I'm going to do a random search, rather than linear, so that everyone has a fair shot during a time-limited search.

Changed the action of &full.
&full Will now give you a "stats menu" of what you can look at.
I've built a rather innefficient method for subdividing the domains, based on what level you want.
For an example machine name of nilwoods.ucs.ualberta.ca...

a &level1 will display stats based on "ca"
a &level2 will display stats based on "ca.ualberta"
a &level3 will display stats based on "ca.ualberta.ucs"
anything else will display stats based on the entire domain, basically mimicking the "old" &full.

Bug in my backup techniques for keeping a busy counter in sync.
Normally, when a file-locking problem causes the counter to "seem like zero", I double-check the count with the "daily stats" file. If that file is busy, I count the number of lines in the "extended log (last n users)". This doesn't work all that well when extended logs aren't being kept (ref per day>100).
For "no extended stats" counters, I only check "daily", and if that doesn't go, I abort the count, and you get the "broke" gif for this one "locked" access.

A Christmas present: Extra logs done. "Top N Users", "Top N Domains", etc.
To use, either go into the Housekeeping, and press buttons, or add a number (like 20) to the end of a leveln, like this:

&level1:10 will show top 10 countries (here is an example)
&level2:15 will show top 15 companies (here is an example)
&level3:15 will show top 15 subdomains (here is an example)
&frequent:20 will show top 25 individuals (here is an example)

I expired all counters that haven't been referenced in a month. That includes the "registered" ones that haven't been referenced in a month, too.

Added a button to the "Inspector" in housekeeping which can be used to re-synchronize the "daily total" count with the "normal" count. This is better than a reset, which wipes out the daily stats. Use this when you feel the count that is being displ ayed is different than what "daily" is reporting. NOTE: This will set both of these numbers to the SAME value, that being the HIGHER of the two. -- Nothing else will be changed.

NEW HONKIN' SERVER added to handle the system load. Site was up to 200,000 big cgi-script hits per day with basically 1 server. Although a little premature to tell, the system load seems to now be much more improved/balanced.
• Update Jan 21: New machine doesn't handle perl "unlink" properly. Broken gifs resulted. I think I've fixed it as of 5pm.

Dec 20/95 Dump:

• Made a modification to the Registration program that lets you "claim"/"inherit" a non-registered counter. It should preserve the previous counts, create date, logs, etc.

• Migrated the Counter program (and its DBM databases) to a bigger server. Caused two hours of down time. Although this was extremely tough, because of the re-fitting of little-endian data onto a big-endian machine, everything should be transparent, but if you notice problems that have cropped up because of this significant change, EMail me. (I'd also be interested to know if you're now getting better performance)
  Sep 29:
  Registration program broke. File & PATH permissions. Fixed around 11:30am.
  Sep 30:
  ARGGH. Some people read too well. If you have references to Geo-counter.new -- get rid of them. Also, working on a fix to the "stats" pages.

Oct 10:

Hey! Someone has pointed out that some browsers are more sticky about what gets used as comments!

Change all occurrences of:

       <!--------- Some comment --------->
to:
       <!-- ======== Some comment ======= -->
  

Sorry about that!

I fiddled with the names of some of the fonts. It was dumb to have some "Mediums" largers than some of the "Bigs". So, some people might have to adjust their font names. I won't do this too often - I just thought this would be better. See http://www.ualberta.ca/htbin/showfonts

Pre-Dec20/95 dump

• Counter has been re-released.
• Several restrictions have been added. (See notes)
• For a background on the progress of these changes, see Geo-counter Progress page

• Change to the algorithm for Histograms to be a little nicer at scaling.
• Most data on server moved to shared space to prepare for move to mutiple-cpu service.
• Tightened a few of the blacklist algorithms.

• Bug fixed: Reporting of advanced stats was being turn off prematurely for registered counters. Should now properly report if RPD<100, rather than 50.
  I had to deal with a couple of really snide people about this, who, rather than saying, "Geo, I'm at 70rpd and registered, and it's not letting me see the advanced stats", I was getting, "Hey. How come I can't see my advanced stats!?" Click here for my response

• Bug fixed: "Reset to ZERO" under registration procedure actually only reset the attributes, but not the count. That part now works.
• Produce better error messages when you are asking for stats on a counter that isn't available.

• Bug fixed: "IGNORE LIST" under housekeeping procedure was not working for a few people. Turns out they were adding unprintable characters at the end of the lists of domain names. (Like using a TAB to go to the next field when the browser can't handle it). This was hard to track down, but I think I've devised a way to ignore those unprintables. --- The real answer was/is to re-type the entries.
• I'm also bowing to pressure (from too many people who can't read), and changing the "Not now-Try Later" GIF to include the fixed times 8am-5pm Edmonton Time, even though that means that if I change the restricted hours they wan't be reflected in the GIF. Ie, so much for staying generic for portability.

Below this line are really old changes

• Expired counters not referenced since May 18
• made the "bad counter reset" a nightly cron-job.

May 15/95 Resetting URLs

• Implemented an automatic "reset" of the URL database.
  • One chance in 200 that the URL list will be reset itself, and lowered the chances that a new URL will "pop" into the list from 10% to 3%, hopefully minimizing the FAQ 2 email questions.

• I'm still tracking down the following problems:
  • Transparent GIF filter causes server to crash occasionally
    I've disable transparency for the time being...
    Chose one of the other fonts for a while, if you don't want to "put up with the ugliness"
  • Symptom: A few *HEAVILY* used counters keep getting reset to 1
    File locking thru Perl isn't as good as advertised, and I'm struggling with making it better. Heavily used (> 20 access per minute) pages are the only counters prone to this behaviour.
    ALL YOUR DATA is *still there* - you just have to look at it "manually" by using the &all parameter.
    ****Update: May 12**** While I still can't figure out how to solve this, I built a kludge to re-count the log files, so the counters will show up "right" again. I will re-run this script occasionally, to ensure that the counts are right for the high-use people...

• Big Fat Crash Number TWO
  • The whole system fell apart. A victim of success. And too many changes at once (ie gifs)...
  • Probable causes:
    • over 8 hits per second on the server does wonders for log files, locked files, and swap space.
    • Not enough children being forked, or forked-tasks not terminating/expiring fast enough?
  • Server software upgraded to NCSA HTTPD 1.4,
  • 40 Children allowed, and TimeOut set at 5 minutes.
  • Server swap space enlarged to 400MB
  • Moved internally some external routines that used to be called as "system" commands, potentially forking new processes that might have a tendency to (not) die. Others utils are still called the "easy way", and may still hose us.
  • I'm still getting spurious "httpd: recvmsg errorrecvmsg: Interrupted system call" errors.
  • Modifications to the error-checking and debugging have been done, so I can track this stuff a little better the next time it happens --- Yes, I do expect a next time!
  • If at all possible, I'll attempt to throw up the "not avail" gif, when something be broke.
  ... And only 25 messages, telling me it was down.

  Started work: Tuesday 1:30am, finished work: Wednesday 5:30am. I now get to start my regular work day...

GIF IMAGES ARE AVAILABLE!!

Lots of font choices!! About Time!! Hooray!!

Change all references to the program from:


http://www.ualberta.ca/htbin/Geo-counter.xbm?yoursite.yourfile.xbm
to be
http://www.ualberta.ca/htbin/Geo-counter.gif?yoursite.yourfile.gif


After that, by default, you'll get a font that looks like this:

You can change your counter's look by adding the phrase "&font:Size-style" to the end of the request. There are lots of font choices. So, here's what your counter request should look like to get the Medium-green characters:


<img src="http://www.ualberta.ca/htbin/Geo-counter.gif?yoursite.yourfile.gif&font:Medium-green">


Note that it takes longer to load gifs, because there's more info. If you want fast, continue using ".xbm"

• MEANINGFUL NAME CHECKING
  • I've added some error checking when the program encounters a new counter request. If you follow the new MEANINGFUL EXAMPLES page, you can get an idea of what to expect when you encounter an error.

• Time Zone Info Added
  • Added a quick-fix to tell people that times are reported in MDT. A more permanent solution will exist when I build a registration system. See FAQ 4

• CLEANUP & DISK MOVE
  • I've just about run out of disk. I've moved the counters to a new volume, but that's temporary, too! I'm getting about 90 new counters A DAY!!
  • Because of the migration, some of the activity between 8pm-10pm Edmonton Time were lost. Sorry 'bout that.
  • I also expired counters not referenced since 9500420. That made 286 abandoned counters in one month.
  • I (hopefully) corrected a slight problem in analyzing which domain names to reverse.

• CLEANUP DONE
  • "Ownership" URLs erased, but will re-populate themselves as appropriate browers can tell me what the web page was that called the counter. (See FAQ 2 for how/why this happens)
  • Any log file that has not been referenced since 950324 has been removed.
  • Any file with bizarre characters or odd file names was removed. (These include ones that started with a dash, or had a '>' in it). Files with special characters, or stupid names will continue to be periodically eradicated. (See Meaningful Names Examples for more details).

• Reversed domains didn't make any sense for numbers, (duh) , so a hook was put in to preserve numeric ip addresses in "&full" stats. To properly reverse-order the legacy of data, the server was "hiccupping" for about 20 minutes between 2:30 and 3:00pm - some files may have lost a few hits.
• After 12500 hits, one of the counters seemed to reset. This random occurrence might be related to the heavy usage of the file. I've added extra "locking" checks during file-opening procedures to hopefully compensate for this weird thing. (Funny - I thought Perl was supposed to manage these suckers for me)

• Small bug fix lets users use tildes in their chosen log names. Note that this may effect some counters that had special characters (especially backslashes).
• Built a hook to record "first use"/"create date" of the counter. Retrieved only in "log" printouts, currently.

• Modification to filter out "xxxyyy.html#anchor" and imagemap "xxxyyy.blah?123,456" urls from ownership registries. "Owner" databases reset, but should re-fill themselves quickly.

• Changed the "date" reporting so it doesn't take up as much room (or disk space)
• Printouts now contain the URL(s) -if known- that reference the counter
• Format of Printout revised to have a better preformatted appearance
• Add the " "full domain stats"
  option
  This is a new report that shows sorted statistics based on the machine name (by reversed domain name). One stats line per visitor. This is something I've wanted for a while, so it's done. Note, however, you may not want to add this "full view" to the average user. Just know how to reference it yourself, perhaps?
  See this new hunk of html code that you can add to your file to give you "full" stats.

Made a linked synonym of "Geo-counter" to "Geo-counter.xbm"
This is obsolete.
Some browsers can indeed interpret inline X-bitmaps, but instead of
looking at the mime content look at suffix of the path to the script.
Change all references to the program
http://www.ualberta.ca/htbin/Geo-counter
to be
http://www.ualberta.ca/htbin/Geo-counter.xbm
to allow more client browsers to properly view the counter.

Made a hook to track signatures of the web documents that use the Geo-counter program. This hook will record "collisions" but will not police two html documents "sharing" the same log files.

1. Some browsers can't handle XBM graphics.
2. Sending a graphic is slower than sending 3 or 4 digits.
3. The visual "look" of the counter output does not necessarily "fit in nicely" with the rest of your page.

1. I'm thinking of switching to GIF, if I can figure out how to do it. (Done: May 4/95)
2. XBM is a text-only stream. It's quite small, but still hundreds of characters. I'll see what I can do. (Done: May 4/95)
3. I've built the program to be flexible (sort of), and in the future, I'll be giving CHOICES of different charactersets to use. (Done: May 4/95)

This seems to no longer be an FAQ, so it's been moved here.

Would you give me a quick explanation regarding the "Log referenced by URL:" part of the tail log?

I made a point of using the entire path to each document that calls your program because I didn't want other people to be using the same variable (I notice two people seem to be calling the Geo-counter with index.html; I wanted my counter variable to be unique so I used a full path). However, much to my surprise, today when I checked the tail log of one of my pages, I discovered that listed under "Log referenced by URL:" was

http://home.mcom.com/home/welcome.html (the Netscape home page)
Any ideas as to why that showed up as a referenced URL? I'm sort of baffled

Unique names are good. I like unique names.

The "Referenced by: ..." stuff takes a little explaining.

Some, but not all browsers (ie your desktop app, such as Mosaic, WinWeb, Netscape) tell the Web Server (httpd daemon) extra environment variables, one of which is the variable HTTP_REFERER -- this variable tells the httpd what URL asked for the new resource (eg an XBM file). For a little info on this, see http://www.ualberta.ca/WebSupport/DumpEnvironment.html

Because I do not have a registration procedure for this program (hence the "pick your UNIQUE NAME, and be polite"), I have to rely on someone else to provide me with the name of the URL that called the logger. This only happens when *any* user with the HTTP_REFERER visits the page (remember, there are no counter "owners" -- be polite).

I have noticed that some browsers that provide the HTTP_REFERER variable get confused once in a while, and actually don't remember where they truly came from. My suspicion is that it happens when a user calls a page directly from his hotlist, after JUST launching the app.

Oh well. It's just a courtesy piece anyway. I thought it would help to alert any individual that someone else "wasn't being polite", and cloned your counter name.

The files that contain the HTTP_REFERER URLs can be wiped out, and my program will "re-generate" the list, as new visitors arrive. I'm thinking of making this a monthly house-keeping chore.