PowerBASIC Forums
  Programming
  Process Memory Info

Post New Topic  Post A Reply
profile | register | preferences | faq | search

UBBFriend: Email This Page to Someone! next newest topic | next oldest topic
Author Topic:   Process Memory Info
Michael Mattias
Member 		posted January 05, 2004 09:21 AM     Click Here to See the Profile for Michael Mattias     Edit/Delete Message   Reply w/Quote
I have a client for whom I developed an application which reads and reformats a data file.

Pseudo-code:

Load File Data to single dynamic string. File contains variable length delimited records.
Parse file to find each record, store into an Array, resizing array as required
Call a series of data-fixup routines something like


    CALL Fixup1 (Array())
    CALL Fixup2 (Array())
    etc.

Save modified array to fixed-record-length file.

In the Fixup routines, Array() is never resized.

When the files were the size originally specified, (10-30 Mb), no problem.

But due to increased business, files are now in the 60-80 Mb range, and when this program runs customer says system performance is degraded. This is an NT Server running lots of jobs all day long.

We "think" the performance hit is due to the large dynamic allocation of memory. Program obviously allocates (filesize) bytes to load the file data, plus approximately (filesize) bytes to hold the array into which the file is parsed. And while I recently changed to allocate an approximate number of elements for the array 'as soon as possible' (that is, when the file is first loaded), it is possible that at some point during the program run I need to do a REDIM PRESERVE ARRAY(currentsize + more space), which means for a brief moment the RAM requirement is (filesize) + (2 * ArraySize) ... with a 70 Mb input file, about 210 Mb!

We "think" we want to try to reduce the dynamic allocation requirement by using disk instead of RAM, but before we make the investment in this change (not really that much, maybe a half day. Will change to use memory-mapped file object for input, and fixed-length random-access work file for fixup routines), we want to get some idea of the actual usage of RAM by the program.

So....

I found the PSAPI WinAPi functions, especially GetProcessMemoryInfo(), which looks like it returns the info we want.. or at least, as close as we're going to get on a 'live' system.

GetProcessMemoryInfo returns this structure:


TYPE PROCESS_MEMORY_COUNTERS
   cb AS DWORD                           '// Size of the structure, in bytes.
   PageFaultCount AS DWORD               '// Number of page faults.
   PeakWorkingSetSize AS DWORD           '// Peak working set size.
   WorkingSetSize AS DWORD               '// Current working set size.
   QuotaPeakPagedPoolUsage AS DWORD      '// Peak paged pool usage.
   QuotaPagedPoolUsage AS DWORD          '// Current paged pool usage.
   QuotaPeakNonPagedPoolUsage AS DWORD   '// Peak nonpaged pool usage.
   QuotaNonPagedPoolUsage AS DWORD       '// Current nonpaged pool usage.
   PagefileUsage AS DWORD                '// Current space allocated for the pagefile.
                                         '// Those pages may or may not be in memory.
   PeakPagefileUsage AS DWORD            '// Peak space allocated for the pagefile.
END TYPE

I've been thru the SDK a couple of times, and still can't figure out which of these numbers - if any - are the numbers which tell us "yes, this change has / has not reduced the dynamic allocation requirements of this application."

I have to believe those "peak" members are the key values, but I'm not sure.

Can anyone point me to a good reference, or explain which numbers are important? I'm going to report everything just before end of job. I just don't know which of those numbers is meaningful.

Thanks,

------------------
Michael Mattias
Tal Systems Inc.
Racine WI USA
mmattias@talsystems.com
www.talsystems.com

IP: Logged

Michael Mattias
Member 		posted January 11, 2004 10:03 AM     Click Here to See the Profile for Michael Mattias     Edit/Delete Message   Reply w/Quote
If I may ask a favor??
Since I use Win/98, I was unable to test this on a machine where it should work..

Could people with Win NT, Win2K and Win Xp please test this on those O/S's please?

If it works out, I think this will produce something useful for everyone...

Thanks,


' FILE : Process_memoryInfo.bas
' File to be #INCLUDEd in programs to get memory usage info about the program
' PSAPI header conversions Courtesy: Jose Roca 11/2002 at
' http://www.powerbasic.com/support/forums/Forum7/HTML/001630.html 
' The PSAPI functions are only supported on platform 5 machines. THis code detects OS and acts accordingly
' This file contains only the DECLARES, UDTs and functions necessary to
' support the getProcessMemoryInfo function
' 1/11/04
' Michael Mattias Racine WI
' Tested OK on Win/98 1/11/04 (not that that is a terribly useful test.)
' compiier: PB.WIn v 7.02
' Win32API.INC   May 9 2002

' STRUCTURE RETURNED BY GetProcessMemoryInfo
'  The "WorkingSetSize" members are the best indicator of a process' memory usage.
TYPE W_PROCESS_MEMORY_COUNTERS
   cb                         AS DWORD   '// Size of the structure, in bytes.
   PageFaultCount             AS DWORD   '// Number of page faults.
   PeakWorkingSetSize         AS DWORD   '// Peak working set size.
   WorkingSetSize             AS DWORD   '// Current working set size.
   QuotaPeakPagedPoolUsage    AS DWORD   '// Peak paged pool usage.
   QuotaPagedPoolUsage        AS DWORD   '// Current paged pool usage.
   QuotaPeakNonPagedPoolUsage AS DWORD   '// Peak nonpaged pool usage.
   QuotaNonPagedPoolUsage     AS DWORD   '// Current nonpaged pool usage.
   PagefileUsage              AS DWORD   '// Current space allocated for the pagefile.
                                         '// Those pages may or may not be in memory.
   PeakPagefileUsage          AS DWORD   '// Peak space allocated for the pagefile.
END TYPE

#IF 0
' ORIGINAL DECLARE
DECLARE FUNCTION GetProcessMemoryInfo LIB "PSAPI.DLL" ALIAS "GetProcessMemoryInfo" ( _
        BYVAL hProcess AS DWORD, _
        ppsmemCounters AS PROCESS_MEMORY_COUNTERS, _
        BYVAL cb AS DWORD _
) AS LONG
#ENDIF
' DECLARE FOR USE WITH CALL DWORD
DECLARE FUNCTION W_GetProcessMemoryInfo _
      ( BYVAL             hProcess AS DWORD, _
        ppsmemCounters             AS W_PROCESS_MEMORY_COUNTERS, _
        BYVAL             cb       AS DWORD _
      ) AS LONG
       
#IF NOT %DEF(%WINAPI)
  #INCLUDE "WIN32API.INC"
#ENDIF
$PSAPI_LIBNAME               =   "PSAPI.DLL"
$PSAPI_GETPROCESSMEMORYINFO  = "GetProcessMemoryInfo"
TYPE FormatWPMCType
    Statistic       AS STRING *  40
    Value           AS STRING *  16
    eol             AS STRING *   2   ' end of line
END TYPE

FUNCTION Formatted_WPMC (WPMC AS W_PROCESS_MEMORY_COUNTERS) AS STRING
    LOCAL w AS STRING, mask AS STRING, FW AS FormatWpmcType
    
    mask   = "* #,"      ' format for integers
    w      = ""
    
    FW.Eol               =  $CR & $LF           ' or $CRLF if sufficient compiler version
    
    LSET FW.Statistic    = "Page Fault Count"
    RSET FW.Value        =  FORMAT$(WPMC.PageFaultCount, mask)
    w                    =  w & PEEK$(VARPTR(FW), SIZEOF(FW))
    
    LSET FW.Statistic    =  "Peak Working Set Size"
    RSET FW.Value        =  FORMAT$(WPMC.PeakWorkingSetSize, mask)
    w                    =  w & PEEK$(VARPTR(FW), SIZEOF(FW))
    
    LSET FW.Statistic    =  "Current Working Set Size"
    RSET FW.Value        =  FORMAT$(WPMC.WorkingSetSize, mask)
    w                    =  w & PEEK$(VARPTR(FW), SIZEOF(FW))
    
    LSET FW.Statistic    =  "Quota Peak Paged Pool Usage"
    RSET FW.Value        =  FORMAT$(WPMC.QuotaPeakPagedPoolUsage, mask)
    w                    =  w & PEEK$(VARPTR(FW), SIZEOF(FW))
    
    LSET FW.Statistic    =  "Current Quota Paged Pool Usage"
    RSET FW.Value        =  FORMAT$(WPMC.QuotaPagedPoolUsage, mask)
    w                    =  w & PEEK$(VARPTR(FW), SIZEOF(FW))
    
    LSET FW.Statistic    =  "Quota Peak Non-Paged Pool Usage"
    RSET FW.Value        =  FORMAT$(WPMC.QuotaPeakNonPagedPoolUsage, mask)
    w                    =  w & PEEK$(VARPTR(FW), SIZEOF(FW))
    LSET FW.Statistic    =  "Current Quota Non-Paged Pool Usage"
    RSET FW.Value        =  FORMAT$(WPMC.QuotaNonPagedPoolUsage, mask)
    w                    =  w & PEEK$(VARPTR(FW), SIZEOF(FW))
    
    LSET FW.Statistic    =  "Peak Page File Usage"
    RSET FW.Value        =  FORMAT$(WPMC.PeakPageFileUsage, mask)
    w                    =  w & PEEK$(VARPTR(FW), SIZEOF(FW))
    LSET FW.Statistic    =  "Current Page File Usage"
    RSET FW.Value        =  FORMAT$(WPMC.PageFileUsage, mask)
    w                    =  w & PEEK$(VARPTR(FW), SIZEOF(FW))
    FUNCTION             =  RTRIM$(W, ANY CHR$(13,10))   'remove trailing CRLF
    
END FUNCTION

' returns: TRUE: S has CRLF delimited process memory info, FALSE S contains messages why it doesn't
FUNCTION GetProcessMemoryStatistics (S AS STRING) AS LONG
    
    LOCAL hProcess AS DWORD,  lret AS LONG, cb AS LONG, isP5 AS LONG
    LOCAL WPMC     AS W_PROCESS_MEMORY_COUNTERS
    LOCAL OSV AS OSVERSIONINFO
    LOCAL szLibName    AS ASCIIZ * %MAX_PATH, szProcName AS ASCIIZ * %MAX_PATH
    LOCAL hLIB         AS LONG,  dwAddr AS DWORD
    
    FUNCTION                  =  %FALSE         ' don't set to true until we have the info
    s                         = ""
    
    ' get the OS Version and decide if we are on a platform 5 machine
    OSV.dwOsVersionInfoSize   =  SIZEOF(OSV)
    lRet                      =  GetVersionEx (OSV)
    IF ISTRUE lRet THEN
        IsP5                  =  (OSV.dwPlatformID  = %VER_PLATFORM_WIN32_NT)
    ELSE
        S = "Can't get operating system version.. this is really bad!"
        EXIT FUNCTION
    END IF
    
    ' if this is a platform 5 machine..
    
    ' Lie about OS for test on Win/98, should return unable to load library, it does.
    ' isP5     = %TRUE
    
    IF ISTRUE  isP5 THEN
        ' Get the address of the function in PSAPI.DLL
        szLibName      = $PSAPI_LIBNAME
        szProcName     = $PSAPI_GETPROCESSMEMORYINFO
        
        hLib           = LoadLibrary (szLibName)
        IF ISTRUE  hLIB THEN
            dwAddr     =  GetProcAddress (hLib, szProcName)
            IF ISTRUE dwAddr THEN
                hProcess      = GetCurrentProcess
                cb            = SIZEOF(WPMC)
                ' call the function
                CALL DWORD dwAddr USING W_GetProcessMemoryInfo (hPRocess, WPMC,cb) TO lret
                IF ISTRUE lret THEN     ' function succeeded, format the WPMC structure
                    S     = Formatted_WPMC (WPMC)
                    FUNCTION   = %TRUE
                ELSE
                    S     = "GetProcessMemoryInfoFailed"
                END IF
            ELSE
                S         = "Can't Get Address for " & szProcName
            END IF
            FreeLibrary      hLib
        ELSE
            S             = "Can't load library " & szLibName
        END IF
        
     ELSE
            s             = "Process Memory Information N/A on this operating system."
            #IF 0
            ' for test of formatting only on Win 9x..
            WPMC.PageFaultCount              = 1234567890
            WPMC.PeakWorkingSetSize          = 1234567891
            WPMC.WorkingSetSize              = 1234567892
            WPMC.QuotaPeakPagedPoolUsage     = 1234567893
            WPMC.QuotaPagedPoolUsage         = 1234567894
            WPMC.QuotaPeakNonPagedPoolUsage  = 1234567895
            WPMC.QuotaNonPagedPoolUsage      = 1234567896
            WPMC.PeakPageFileUsage           = 1234567897
            WPMC.PageFileUsage               = 1234567898
            s             = Formatted_WPMC (WPMC)
            #ENDIF
     END IF
     
                
END FUNCTION
                
FUNCTION PBMAIN () AS LONG
    
  LOCAL s AS STRING, l AS LONG
  
  
  ' typically you would call this as the last action of WinMAin...
  L = GetProcessMemoryStatistics (s)
  
  MSGBOX S,,"Process Memory Statistics"
    
    
END FUNCTION
    
' // END OF FILE

IP: Logged

John Revill
Member 		posted January 11, 2004 10:26 AM     Click Here to See the Profile for John Revill     Edit/Delete Message   Reply w/Quote
Windows 2000 Server, Service Pack 4, 512 MB
  
  ---------------------------
  Process Memory Statistics
  ---------------------------
  Page Fault Count                                     354
  Peak Working Set Size                          1,458,176
  Current Working Set Size                       1,458,176
  Quota Peak Paged Pool Usage                       14,988
  Current Quota Paged Pool Usage                    14,676
  Quota Peak Non-Paged Pool Usage                    2,544
  Current Quota Non-Paged Pool Usage                 1,668
  Peak Page File Usage                             380,928
  Current Page File Usage                          380,928
  ---------------------------
  OK
  ---------------------------
 
 
Windows XP, Service Pack 1, 512 MB
 
  ---------------------------
  Process Memory Statistics
  ---------------------------
  Page Fault Count                                     331
  Peak Working Set Size                          1,359,872
  Current Working Set Size                       1,359,872
  Quota Peak Paged Pool Usage                       17,136
  Current Quota Paged Pool Usage                    16,924
  Quota Peak Non-Paged Pool Usage                    2,288
  Current Quota Non-Paged Pool Usage                 1,320
  Peak Page File Usage                             356,352
  Current Page File Usage                          356,352
  ---------------------------
  OK   
  ---------------------------

------------------

[This message has been edited by John Revill (edited January 11, 2004).]

IP: Logged

Michael Mattias
Member 		posted January 11, 2004 11:30 AM     Click Here to See the Profile for Michael Mattias     Edit/Delete Message   Reply w/Quote
Thank you... so it "appears" that I didn't do anything totally dopey...

But now it becomes obvious from the statistics that I should have put in some memory allocation function to look at Peak v Current

So what if we make the PbMain function look like this...


FUNCTION PBMAIN () AS LONG
    
  LOCAL s AS STRING, l AS LONG, Z() AS STRING * 1024
  ' allocate   50 mb of memory and get statistics..
    REDIM  Z(50 * 1024)
    L = GetProcessMemoryStatistics (s)
    MSGBOX S,,"Process Memory Statistics with Array Allocated"
  ' De-allocate the memory... 
    ERASE Z()
  ' ... and get 'end of job' statistics.   
  ' We should see 'peak' numbers much less than 'current' numbers
  ' (and somewhere approximately 50 Mb of difference)
    L = GetProcessMemoryStatistics (s)
  
   MSGBOX S,,"Process Memory Statistics at End of Job"
    
END FUNCTION

Thanks,
MCM

IP: Logged

Steve Matthews
Member 		posted January 11, 2004 11:40 AM     Click Here to See the Profile for Steve Matthews     Edit/Delete Message   Reply w/Quote 
(compiled with #debug error on, #register none)
Modified PBmain code used

W2K Workstation, SP4, 786mb
---------------------------
Process Memory Statistics with Array Allocated
---------------------------
Page Fault Count                                  13,170
Peak Working Set Size                         54,005,760
Current Working Set Size                      54,005,760
Quota Peak Paged Pool Usage                       15,052
Current Quota Paged Pool Usage                    14,740
Quota Peak Non-Paged Pool Usage                    1,968
Current Quota Non-Paged Pool Usage                 1,720
Peak Page File Usage                          52,871,168
Current Page File Usage                       52,871,168
---------------------------
---------------------------
Process Memory Statistics at End of Job
---------------------------
Page Fault Count                                  13,285
Peak Working Set Size                         54,439,936
Current Working Set Size                       1,970,176
Quota Peak Paged Pool Usage                       15,852
Current Quota Paged Pool Usage                    14,812
Quota Peak Non-Paged Pool Usage                    1,968
Current Quota Non-Paged Pool Usage                 1,928
Peak Page File Usage                          53,010,432
Current Page File Usage                          552,960
---------------------------

XP Workstation, SP1, 786mb, Dual CPU 
---------------------------
Process Memory Statistics with Array Allocated
---------------------------
Page Fault Count                                  13,230
Peak Working Set Size                         54,239,232
Current Working Set Size                      54,239,232
Quota Peak Paged Pool Usage                       17,988
Current Quota Paged Pool Usage                    17,988
Quota Peak Non-Paged Pool Usage                    2,416
Current Quota Non-Paged Pool Usage                 1,560
Peak Page File Usage                          52,871,168
Current Page File Usage                       52,871,168
---------------------------
---------------------------
Process Memory Statistics at End of Job
---------------------------
Page Fault Count                                  13,422
Peak Working Set Size                         54,894,592
Current Working Set Size                       2,441,216
Quota Peak Paged Pool Usage                       19,096
Current Quota Paged Pool Usage                    18,956
Quota Peak Non-Paged Pool Usage                    2,656
Current Quota Non-Paged Pool Usage                 1,800
Peak Page File Usage                          53,047,296
Current Page File Usage                          593,920
---------------------------

[This message has been edited by Steve Matthews (edited January 11, 2004).]

IP: Logged

Raymond King
Member 		posted January 11, 2004 01:59 PM     Click Here to See the Profile for Raymond King     Edit/Delete Message   Reply w/Quote
Michael,

What about using a Memory map file you think that would help?

Enjoy
Ray

------------------
Raymond L. King
Custom Software Designers.

http://www.csdsoft.com
support@csdsoft.com

IP: Logged

Michael Mattias
Member 		posted January 11, 2004 04:16 PM     Click Here to See the Profile for Michael Mattias     Edit/Delete Message   Reply w/Quote
Aha! So it looks like Windows sends the big allocation directly to the swap file instead of sucking up current RAM.. actually makes sense, since we never modified any of the 50 Mb we allocated.

This is going to be a fun thing to see what happens when I change the "real" application....

>What about using a Memory map file you think that would help?

That is exactly what I am planning. But I wanted to get this 'report memory statistics' thing working first, so we have a reasonable "before" picture. (I am putting the statistics thing in the existing software, and client will run it against "big file" keep the numbers.)

How I am going to change the application is...
1. Instead of loading the file data to a string and parsing it, I will memory map the file and parse the mapped view. (I'm already parsing via a pointer variable anyway).
2. Instead of creating the array and loading data to it, I am going to write each array element to a disk file.
3. When the input has been parsed and all the output written to the work file, I will memory map that file, and do a "REDIM array(size) AT (pointer returned by MapViewOfFile)"

The rest of the code remains the same.

I should have "before" and "after" info with "live" data later this week.

This should be very interesting!

(I'll also clean up the above demo and post it in the source code section so others can find it and use it.

MCM

IP: Logged

Michael Mattias
Member 		posted January 13, 2004 07:16 AM     Click Here to See the Profile for Michael Mattias     Edit/Delete Message   Reply w/Quote
Bravely, he posted the actual 'before' figures from the Real Live Customer system (Windows/2000) ...knowing full well if his proposed change doesn't do anything dramatic he's going to look like a dope...


 ========================================
    MEMORY STATISTICS FOR THIS PROCESS
=======================================
Page Fault Count                               1,176,265
Peak Working Set Size                        154,173,440
Current Working Set Size                      81,588,224
Quota Peak Paged Pool Usage                       15,036
Current Quota Paged Pool Usage                    14,832
Quota Peak Non-Paged Pool Usage                    3,096
Current Quota Non-Paged Pool Usage                 2,188
Peak Page File Usage                         223,109,120
Current Page File Usage                      149,454,848

      *** END OF RUN REPORT ***

IP: Logged

Michael Mattias
Member 		posted January 13, 2004 07:37 AM     Click Here to See the Profile for Michael Mattias     Edit/Delete Message   Reply w/Quote
Cleaned up demo, tested only PB/Windows but should work as is with PB/CC:

http://www.powerbasic.com/support/forums/Forum7/HTML/002201.html

MCM

IP: Logged

Paul Dwyer
Member 		posted January 14, 2004 09:29 AM     Click Here to See the Profile for Paul Dwyer     Edit/Delete Message   Reply w/Quote
Michael,

can't you stream the file so that you don't put so much data in memory? what happens if the file is a Gig or just too big to keep in memory some time in the future. Since it happened once sure it can happen again?

If the records require processing 'vertically' then it might be a problem but otherwise wouldn't it scale better (and hog less resources on this busy box) by reading say 100 records, processing them and then reading 100 more etc etc?

This application seems like each function does it's job once them passes the buck, perhaps a more flowing approach would work better?

sort of depends on what "fixupX" does I suppose

------------------
Paul Dwyer
Network Engineer
Aussie in Tokyo

IP: Logged

Michael Mattias
Member 		posted January 15, 2004 07:27 AM     Click Here to See the Profile for Michael Mattias     Edit/Delete Message   Reply w/Quote
>can't you stream the file ...

Actually, yes, I could have... well, if not 'streamed' I could parse part of the file, process it, and continue with the next part of the file. But as I pointed out, when the file sizes were "as advertised" I did not see any need to do so.

As far as the 'passing the buck' to multiple 'fixup' function things...
Client has a long history of changing one particular fixup, adding more fixups and deleting still other fixups after initial specifications. I chose to use the approach above specifically on the assumption client would make changes after I delivered. (FWIW, he did) (several times). In other words, I put "maintainability" very, very high on the list of "considerations when designing this software."

IP: Logged

Michael Mattias
Member 		posted March 26, 2004 11:42 AM     Click Here to See the Profile for Michael Mattias     Edit/Delete Message   Reply w/Quote
Client ran the revised program, same input file... here are the results

Stats for the revised program:


 ========================================
    MEMORY STATISTICS FOR THIS PROCESS
=======================================
Page Fault Count                                  36,796
Peak Working Set Size                         76,447,744
Current Working Set Size                       1,941,504
Quota Peak Paged Pool Usage                       87,624
Current Quota Paged Pool Usage                    14,944
Quota Peak Non-Paged Pool Usage                    3,044
Current Quota Non-Paged Pool Usage                 2,084
Peak Page File Usage                             647,168
Current Page File Usage                          610,304

      *** END OF RUN REPORT ***

Still not really sure what this means.. However, I am sure that when customer said,"this is fantastic!" whatever it means is good.


IP: Logged

All times are EasternTime (US)

next newest topic | next oldest topic

Administrative Options: Close Topic | Archive/Move | Delete Topic
Post New Topic  Post A Reply
Hop to:

Contact Us | PowerBASIC BASIC Compilers

Copyright © 1999-2005 PowerBASIC, Inc. All Rights Reserved.


Ultimate Bulletin Board 5.45c