Gabriella Davis June 1 2009 04:21:18 PM
This a long long post but I think if you're working with Server Load or planning to you might get something from it...
I was doing some work recently assessing server load on new hardware. This was actually the first time I've worked with the 8.5 Server.Load tools although I had worked with earlier ones. For those of you that haven't worked with it, the Server Load utility runs on a client machine and uses scripts to simulate various user activity against a server, allowing you to gauge if the servers you are intending to use are fit for purpose. The technology is pretty cool and inherits much from previous NotesBench tools.
For this exercise I had 2 brand new servers which were going to be clustered together to handle Notes mail load for 2000 users, evenly balanced across both servers. Since the Server Load documentation states that a single client can only handle up to 1500 separate threads (each one emulating a user activity) I configured 3 separate client machines and a 4th machine for rolling up the data as it was generated by the other three.
The first step, running the agent to create person documents for each user in the empty directory, went fine. I now had a directory with 2000 person documents in it. What I didn't have was any mail files or configuration in the person documents for them. That should be easy. The Server Load comes with 'initialisation scripts' that create mail files for each user and modify the person documents accordingly. There is both an initialisation script for Notes 8 and one for a Cluster environment, that will create the mail file and its cluster mate on the secondary server. Since we are testing a cluster that's what I decided to use.
Client 1 was set up for 650 threads on Server 1
Client 2 for 650 on Server 1
Client 3 for 700 on Server 2
What I should have ended up with was 1300 person documents pointing to mail files on Server 1 and 700 person documents pointing to mail files on Server 2. In addition I should have had 1300 mail files on Server 1 and 700 stubs replicating from Server 2 and vice versa on Server 2.
The first thing I did was run a small test for 10 users on each Client and that's when I discovered that although the Cluster Initialisation script
does create a mail file replica on the secondary server, it sets the ACL of the mail file it creates to just the primary server and the user name (regardless of the ACL in the template) so the cluster replicas can't replicate, they have no access at all to the source databases!. At this point yes I could use Full Access Administration to modify the ACLs of the original databases as they are created but that defeated the purpose of what I was trying to do so instead I reverted to using the N8 Initialisation scripts to create the mail files on the primary server only and planned to replicate those to the secondary server when the scripts were finished. After all I wasn't testing the effective of the scripts, they were just a means to an end.
Now we get to the 2nd reason for not using the cluster script, the options for configuring the N8 Initialisation scripts are much more extensive than those for the Cluster Initialisation scripts. The Cluster scripts only allow you to choose the number of messages you want and their size, the N8 scripts allow you to configure % of Calendar appointments, % with attachments etc. Much more useful for building mail files we are going to test against.
So off I go to start the 3 clients running the N8 scripts and all appears fine. For an hour or so. Then Server Load just disappears. Poof. Gone. No error. No NSD. Nothing. Poof! On all 3 clients at about the same time. I try again and again and again. Each time the same result. Eventually I find a paper published in Feb 2009 by IBM and written by Maureen P Gerlofs and Jim Powers that says on Page 21 "
NOTE: You should not issue this test for more than 300 users at a time;" - Good to know, although not so much of a 'note in italics' as a 'requirement' and not one that appears in the Domino Admin help. So the lesson here is that despite the documentation saying 1500 threads by client, if you're running initialisation scripts that's actually 300. I then tried running the scripts multiple times in batches of 300 threads each time and hey presto! it worked.
After all of that the actual user emulation scripts (N8 mail in this case) worked like a charm, running 650 threads each on all 3 clients and together with the nmon tool (which is documented in the white paper but not in the Domino Admin help) gave same great data. In short I recommend Server Load highly, I just recommend you give the Domino Admin help a miss and use t
his white paper instead :-)
Now to open some PMRs......