From guest Fri Feb 4 08:32:10 1994 Date: Fri, 4 Feb 94 11:33:51 EST From: jfr@cae.ca (Jean-Francois Richard) Message-Id: <9402041633.AA01738@cae.ca> To: info-performer@sgi.sgi.com Subject: Perf. 1.0 memory usage Status: OR Somebody just pointed out to me that my Performer 1.0 application appears to have a size of about 70,000 pages (that's 280 Mb!) according to the ps and top commands. That is very interesting (and potentially dangerous) since the machine it is running on only has 128 Mb of combined physical and virtual memory. In real life, the program never seems to actually use more than 15 to 20 Mb. The memory appears to get reserved when I call pfInit(). Does Performer need to reserve that much memory? Is there a chance that it might try to actually use it and run out of swap space? Is there a way to control Performer's appetite so that the other users of the computer won't freak out when they look at my processes? ---------------------------------------------------------- jfr@cae.ca J.F. Richard CAE Electronics Ltd. Montreal, Canada ----------------------------------------------------------- From guest Fri Feb 4 09:41:23 1994 From: "Michael Jones" Message-Id: <9402040941.ZM12430@babar.asd.sgi.com> Date: Fri, 4 Feb 1994 09:41:01 -0800 In-Reply-To: jfr@cae.ca (Jean-Francois Richard) "Perf. 1.0 memory usage" (Feb 4, 11:33am) References: <9402041633.AA01738@cae.ca> X-Mailer: Z-Mail (3.1b.0 21jan94 MediaMail) To: jfr@cae.ca (Jean-Francois Richard), info-performer@sgi.sgi.com Subject: Re: Perf. 1.0 memory usage Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 Status: OR On Feb 4, 11:33am, Jean-Francois Richard wrote: > Subject: Perf. 1.0 memory usage : :Somebody just pointed out to me that my Performer 1.0 application :appears to have a size of about 70,000 pages (that's 280 Mb!) :according to the ps and top commands. That is very interesting :(and potentially dangerous) since the machine it is running on :only has 128 Mb of combined physical and virtual memory. In real :life, the program never seems to actually use more than 15 to 20 Mb. : :The memory appears to get reserved when I call pfInit(). Does Performer :need to reserve that much memory? Is there a chance that it might try :to actually use it and run out of swap space? Is there a way to :control Performer's appetite so that the other users of the computer :won't freak out when they look at my processes? The default allocation is 256MB. This issue has been dealt with in the upcoming 1.2 release of IRIS Performer with the new pfSharedArenaSize() function: just call it before your pfInit() call. Here's the reference page for the new functions (also pfSharedArenaBase() can be handy too). pfSharedMem(3pf) Silicon Graphics pfSharedMem(3pf) NAME pfInitArenas, pfGetSharedArena, pfGetSemaArena, pfTmpDir, pfGetTmpDir, pfSharedArenaSize, pfGetSharedArenaSize, pfSharedArenaBase, pfGetSharedArenaBase - Shared Memory Functions C SPECIFICATION #include long pfInitArenas(void); void * pfGetSharedArena(void); usptr_t * pfGetSemaArena(void); void pfTmpDir(char *dirname); const char* pfGetTmpDir(void); void pfSharedArenaSize(ulong size); ulong pfGetSharedArenaSize(void); void pfSharedArenaBase(void *base); void * pfGetSharedArenaBase(void); DESCRIPTION pfInitArenas creates a shared memory arena that can be used to allocate memory, locks and semaphores from (see pfMalloc, pfNewSema, pfNewLock). This function is called by pfInit so it is not necessary to call it when using libpf functions. Calling pfInitArenas again after using pfBlist, pfBpipe, pfQueue, pfNewLock, pfNewSema will have adverse affects due to the resetting of internal data structures. pfInitArenas uses usinit to create arenas for both locks and shared memory. These arenas are reflected in the Unix file system. Where these files are created is important. There must be as much space available to the file as is allocated for the arenas. 16K bytes are allocated for the locks and semaphores, which supports approximately 4K locks. pfTmpDir and the environment variablle PFTMPDIR control where pfInitArenas creates its arenas. If neither is specified, the semaphore arena is created in /usr/tmp and the shared memory arena is created in /dev/zero (swap space). If a temporary directory is specified, then files are created in that directory for the shared memory and semaphore arenas. Both these files are unlinked so that they are removed from the file system when the process terminates. Once created, these arenas cannot grow, so Performer tries to create a large (256MB) arena. For shared memory, swap space, the default, is usually preferable to using a file in a directory specified by PFTMPDIR. Using an actual file is slightly slower at allocation time and requires actual disk space for extending the file length to equal the amount of memory allocated (pfMalloc) from the arena. Swap space under 4.0.5 is not resevered or allocated until pfMalloc, so Performer can create a large arena without tying up actual swap space until it's used by pfMalloc. But under 5.0.1/5.1, arena space needs to be reserved up front. In this case, Performer limits the size of its shared memory arena to a fraction of the currently available swap space. This may cause some moderate or large applications to fail during pfMalloc. To avoid this problem under 5.0.1/5.1, increase the virtual size of swap space by creating a zero length file called /swap and adding a line to /etc/fstab: /swap swap swap vlength=5000000 and reboot the system. Under 5.1.2, this is no longer necessary since Performer uses a new IRIX mechanism for reserving swap space as the arena usage grows. pfSharedArenaSize can be used to override the arena size that Performer uses by default (256MB). size specifies the desired size in bytes. pfGetSharedArenaSize returns the arena size in bytes. pfSharedArenaBase sets the base address for the mapping of the shared memory arena. Normally, IRIX chooses the base address. This is useful if the application needs closer control over the layout of virtual address space, e.g. to avoid conflicts with other mappings. pfGetSharedArenaBase returns the base address for the arena. pfGetSemaArena returns a handle to the lock arena and pfGetSharedArena returns a pointer to the shared memory arena. This pointer cannot be used directly, only as an argument to pfMalloc. pfGetTmpDir returns the temporary directory set using pfTmpDir. NOTES These arenas can only be used by related processes. Meaning those processes that share a common heritage. Use pfDataPool for sharing memory between unrelated processes. pfInitArenas should be called before any other forks or sprocs are made. SEE ALSO acreate, usinit, pfMalloc, pfFree -- Be seeing you, mtj@sgi.com 415.390.1455 M/S 7L-590 Michael Jones Silicon Graphics, Advanced Graphics Division 2011 N. Shoreline Blvd., Mtn. View, CA 94039-7311 From guest Fri Feb 4 10:16:57 1994 Message-Id: <9402041816.AA11146@surreal.asd.sgi.com> To: jfr@cae.ca (Jean-Francois Richard) Cc: info-performer@sgi.sgi.com Subject: Re: Perf. 1.0 memory usage In-Reply-To: Your message of "Fri, 04 Feb 94 11:33:51 EST." <9402041633.AA01738@cae.ca> Date: Fri, 04 Feb 94 10:16:25 -0800 From: Jim Helman Status: OR Somebody just pointed out to me that my Performer 1.0 application appears to have a size of about 70,000 pages (that's 280 Mb!) according to the ps and top commands. That is very interesting (and potentially dangerous) since the machine it is running on only has 128 Mb of combined physical and virtual memory. In real life, the program never seems to actually use more than 15 to 20 Mb. With Performer 1.0 on IRIX 4 and Performer 1.2 on IRIX 5.2, this large initialization is not a problems since the OS allocates space on demand without any pre-reservation. However with Performer 1.1 on IRIX 5.X, even though OS does not allocate the space until used, it does pre-reserve that much swap space. This can be bad. Performer will try to find a reasonable initialization size, however sometimes it may reserve too much and keep you from running other large programs. In this event, set the PFTMPDIR to a directory with a lot of space, e.g. /usr/tmp. Load times will be slower than when using swap, but you won't have swap space problems. rgds, -jim helman jimh@surreal.asd.sgi.com 415/390-1151 From guest Fri Feb 4 11:27:38 1994 Date: Fri, 4 Feb 94 14:22:29 -0500 From: lbatayt%r2d2@relay.nswc.navy.mil (Lou Batayte) Message-Id: <9402041922.AA01366@r2d2> To: info-performer@sgi.sgi.com Subject: 1.x memory usage Status: OR I just got a copy of the NASA/COSMIC FAST visualization code. In the initialization for that package they tell the installer to change the variable in the kerne,l "int availsmem_accounting = 0" to "int availsmem_accounting = 1". They further comment that this may cause some programs to "break" that don't care if the memory they have allocated is actually available. Will this mod to the kernel cause performer to break? From guest Fri Feb 4 12:38:18 1994 Message-Id: <9402042023.AA12222@surreal.asd.sgi.com> To: lbatayt%r2d2@relay.nswc.navy.mil (Lou Batayte) Cc: info-performer@sgi.sgi.com Subject: Re: 1.x memory usage In-Reply-To: Your message of "Fri, 04 Feb 94 14:22:29 EST." <9402041922.AA01366@r2d2> Date: Fri, 04 Feb 94 12:23:36 -0800 From: Jim Helman Status: OR Performer prefers availsmem_accounting=0 (note that the variable goes away in 5.2). Having availsmem_accounting=1 could cause the problems I described in the previous message regarding Performer reserving too much swap space for other processes to run. I would leave availsmem_accounting=0. I'm not familiar with how FAST allocates memory, however, I would expect at worst that when FAST runs out of memory, it will dump core rather than print out an error message and exit. One of the authors of FAST was a strong advocate for strict memory accounting (up front rather than when used). I forgot to mention in my previous mail, that under late 5.1 and 5.2 you can still turn off availsmem accounting by adding a large section of "virtual swap" To do this: su # chkconfig vswap on # edit /etc/config/vswap.options The same thing can be achieve by creating a zero length file /somefile and putting a line in your /etc/fstab or interactively invoking the swap(1) (swap -a -v 999999 /somefile) command. rgds, -jim helman jimh@surreal.asd.sgi.com 415/390-1151