Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: ISAPI DLL memory leaks


This question is not answered. Helpful answers available: 1. Correct answers available: 1.


Permlink Replies: 8 - Last Post: Jan 22, 2016 9:15 AM Last Post By: Brian Gochnauer
Brian Gochnauer

Posts: 6
Registered: 10/26/00
ISAPI DLL memory leaks  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 1, 2016 8:06 AM
I've got an application written in XE4, i've been struggling with on memory leaks for 5 years now.
It has 300k clients running about 9 million hits a day on the web servers running a datasnap ISAPI dll.
It is run through an F5 load balancer to two IIS 7.0 servers to an SQL Server 2008 back-end of 180 GB database.

At the webservers' busiest time it only takes about 10 minutes (with 10 worker threads) for each IIS server to hit the memory limit (set to 512MB per worker) and then IIS recycles the worker thread.
Changing any of these parameters of worker threads/ memory limits does not significantly change the failures or leaking rate of memory.

When IIS recycles the workers it causes several errors on the client side from terminated sessions/connections.
There is NO demos or examples of ISAPI using SQL Server or Oracle.

I am using a TDSServerModule in the ISAPI dll to hold the ADO components to connect to the SQL Server 2008 and using TClientDatasets and SQLServerMethods to get to the data to the connections.

If this is just the wrong way to do this the please please please create a detailed YouTube video or document that explains how to create a seriously scalable system with datasnap ISAPI

PLEASE HELP!

Thank you.
Brian

Edited by: Brian Gochnauer on Jan 1, 2016 8:08 AM

Edited by: Brian Gochnauer on Jan 22, 2016 9:13 AM
Danny Heijl

Posts: 8
Registered: 8/28/00
Re: ISAPI DLL memory leaks
Helpful
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 4, 2016 7:31 AM   in response to: Brian Gochnauer in response to: Brian Gochnauer
Brian Gochnauer wrote:
I've got an application written in XE4, i've been struggling with on memory leaks for 5 years now.
It has 300k clients running about 9 million hits a day on the web servers running a datasnap ISAPI dll.
It is run through an F5 load balancer to two IIS 7.0 servers to an SQL Server 2008 back-end of 180 GB database.

You could isolate the main part of you logic that handles a request into a (console) test program or unit test and configure FastMM4 (full version, debug mode) to report any memory leaks.

This would at least show if the leaks are in your own code or not, and if they are, where they occur.

Danny
Brian Gochnauer

Posts: 6
Registered: 10/26/00
Re: ISAPI DLL memory leaks  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 4, 2016 8:07 AM   in response to: Danny Heijl in response to: Danny Heijl
Danny Heijl wrote:
Brian Gochnauer wrote:
I've got an application written in XE4, i've been struggling with on memory leaks for 5 years now.
It has 300k clients running about 9 million hits a day on the web servers running a datasnap ISAPI dll.
It is run through an F5 load balancer to two IIS 7.0 servers to an SQL Server 2008 back-end of 180 GB database.

You could isolate the main part of you logic that handles a request into a (console) test program or unit test and configure FastMM4 (full version, debug mode) to report any memory leaks.

This would at least show if the leaks are in your own code or not, and if they are, where they occur.

Danny

But even something as simple as this leaks a lot of memory

function TServerMethodsUnit1.GetDistrPointInfo(Value1, Value2 :String):TDataset;
begin
  if ADOStoredProc1.Active then ADOStoredProc1.Close;
  with ADOStoredProc1.Parameters  do
    try
      FindParam('@Value1').Value := Value1;
      FindParam('@Value2').Value := Value2;
      ADOStoredProc1.Open;
      Result := ADOStoredProc1;//DataSet;
    except
      on E:Exception do
      begin
        Raise Exception.Create('Exception '+E.ClassName+ ' in ADOStoredProc1: '+e.message);
      end;
    end;
end;
Omar Edgardo Ze...

Posts: 80
Registered: 8/7/07
Re: ISAPI DLL memory leaks [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 5, 2016 9:20 AM   in response to: Brian Gochnauer in response to: Brian Gochnauer
Hi,

Does the memory leak ocurrs just when using it as a ISAPI dll, have you
tested using it as a VCL App or WinService? When using ISAPI, there
should be some considerations on the app, on how IIS works when getting
lots of connections.

How do you handle your data access control components, create a
datamodule per connection or global?

I have a WinService DataSnap App that handles millions requests and I
have no memory issues.(I does not return any cliendatasets, only json)

Omar Zelaya

El 1/1/16 a las 10:08 a.m., Brian Gochnauer escribió:

I've got an application written in XE4, i've been struggling with on memory leaks for 5 years now.
It has 300k clients running about 9 million hits a day on the web servers running a datasnap ISAPI dll.
It is run through an F5 load balancer to two IIS 7.0 servers to an SQL Server 2008 back-end of 180 GB database.

At the webservers' busiest time it only takes about 10 minutes (with 10 worker threads) for each IIS server to hit the memory limit (set to 512MB per worker) and then IIS recycles the worker thread.
Changing any of these parameters of worker threads/ memory limits does not significantly change the failures or leaking rate of memory.
Alexandre Machado

Posts: 1,754
Registered: 8/10/13
Re: ISAPI DLL memory leaks  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 5, 2016 12:42 PM   in response to: Brian Gochnauer in response to: Brian Gochnauer
Hi,

I've done lots of ISAPI development using Delphi (IntraWeb, plain WebBroker, IntraWeb + DataSnap, ISAPI + dbGo, DataSnap only, etc etc...) and I don't recall any memory leaks.

The full version of FastMM4, running in full debug mode is able to give you a precise report of any memory leaks, even when running as an ISAPI app. Create a minimal ISAPI application which does not connect to any DB and does not use DataSnap at all. Add FastMM4 in full debug mode and force a memory leak (this way you will know if FastMM4 is catching it correctly). Check if you get the leak in FastMM4 report. Check if any other memory leaks are present.
Then add other parts one by one: Add the ADO part (without DataSnap), and check if it generates any new memory leaks. Then finally add the DataSnap layer. Double check if your application is using the external Midas.dll or the internal linked MidasLib.dcu.
This should give you a very good idea where the leak is - if it exist at all.

From the top of my head, there is no memory leaks present in a plain ISAPI DLL (without dbGo and DataSnap layers).

Kind regards
Brian Gochnauer

Posts: 6
Registered: 10/26/00
Re: ISAPI DLL memory leaks  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 5, 2016 9:31 PM   in response to: Alexandre Machado in response to: Alexandre Machado
Alexandre Machado wrote:
Hi,

I've done lots of ISAPI development using Delphi (IntraWeb, plain WebBroker, IntraWeb + DataSnap, ISAPI + dbGo, DataSnap only, etc etc...) and I don't recall any memory leaks.

The full version of FastMM4, running in full debug mode is able to give you a precise report of any memory leaks, even when running as an ISAPI app. Create a minimal ISAPI application which does not connect to any DB and does not use DataSnap at all. Add FastMM4 in full debug mode and force a memory leak (this way you will know if FastMM4 is catching it correctly). Check if you get the leak in FastMM4 report. Check if any other memory leaks are present.
Then add other parts one by one: Add the ADO part (without DataSnap), and check if it generates any new memory leaks. Then finally add the DataSnap layer. Double check if your application is using the external Midas.dll or the internal linked MidasLib.dcu.
This should give you a very good idea where the leak is - if it exist at all.

From the top of my head, there is no memory leaks present in a plain ISAPI DLL (without dbGo and DataSnap layers).

Kind regards

I too have written 100,000s of lines of code and yet nobody will demonstrate an actually working dbGo and Datasnap scalable demo that works.
Yes the leaks occur in the standalone EXE versions of the ISAPI implementations.

This problem has been reported many many times; but every QC article magically gets closed and forum posts languishes and never gets answered.
I mean it should not be that difficult to get JUST ONE demo of a scalable dbGO/ISAPI "framework" from Embarcadero or some expert in Datasnap to 'document' the proper design that should work.

Digging into the internals of ISAPI memory management was/should have been done by somebody when the wrote the code to support building ISAPI dlls,
It is not my code; when simple ADO queries leak copious amounts of memory unless I've made errors in the basics of creating an ISAPI dll
if not, then I don't believe I should have to dig into ISAPI debugging (ugh!) and deal with things I can't fix when i do find the problem is inside Embarcadero code base.

Do you have code? Code that you can share that you know/demonstrated that had no leaks; that can do thousands of connections an hour and not leak memory; even if i can't compile completely or run.

Thanks,
Brian
Danny Heijl

Posts: 8
Registered: 8/28/00
Re: ISAPI DLL memory leaks  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 6, 2016 4:31 AM   in response to: Brian Gochnauer in response to: Brian Gochnauer
Brian Gochnauer wrote:

...snip....

Yes the leaks occur in the standalone EXE versions of the ISAPI implementations.

Which suggests that you don't free memory that should be freed. If you use and configure the full FastMM4 in full debug mode and use the debug dcu's the standalone exe fastmm4 log should tell you where the leaks occur.

Danny
Alexandre Machado

Posts: 1,754
Registered: 8/10/13
Re: ISAPI DLL memory leaks  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 10, 2016 4:10 PM   in response to: Brian Gochnauer in response to: Brian Gochnauer
Brian Gochnauer wrote:
Alexandre Machado wrote:
Hi,

I've done lots of ISAPI development using Delphi (IntraWeb, plain WebBroker, IntraWeb + DataSnap, ISAPI + dbGo, DataSnap only, etc etc...) and I don't recall any memory leaks.

The full version of FastMM4, running in full debug mode is able to give you a precise report of any memory leaks, even when running as an ISAPI app. Create a minimal ISAPI application which does not connect to any DB and does not use DataSnap at all. Add FastMM4 in full debug mode and force a memory leak (this way you will know if FastMM4 is catching it correctly). Check if you get the leak in FastMM4 report. Check if any other memory leaks are present.
Then add other parts one by one: Add the ADO part (without DataSnap), and check if it generates any new memory leaks. Then finally add the DataSnap layer. Double check if your application is using the external Midas.dll or the internal linked MidasLib.dcu.
This should give you a very good idea where the leak is - if it exist at all.

From the top of my head, there is no memory leaks present in a plain ISAPI DLL (without dbGo and DataSnap layers).

Kind regards

I too have written 100,000s of lines of code and yet nobody will demonstrate an actually working dbGo and Datasnap scalable demo that works.
Yes the leaks occur in the standalone EXE versions of the ISAPI implementations.

This problem has been reported many many times; but every QC article magically gets closed and forum posts languishes and never gets answered.
I mean it should not be that difficult to get JUST ONE demo of a scalable dbGO/ISAPI "framework" from Embarcadero or some expert in Datasnap to 'document' the proper design that should work.

Digging into the internals of ISAPI memory management was/should have been done by somebody when the wrote the code to support building ISAPI dlls,
It is not my code; when simple ADO queries leak copious amounts of memory unless I've made errors in the basics of creating an ISAPI dll
if not, then I don't believe I should have to dig into ISAPI debugging (ugh!) and deal with things I can't fix when i do find the problem is inside Embarcadero code base.

Do you have code? Code that you can share that you know/demonstrated that had no leaks; that can do thousands of connections an hour and not leak memory; even if i can't compile completely or run.

Thanks,
Brian

First, scalability has nothing to do with memory leaks necessarily. Even applications with zero memory leaks may not scale well. Also, nothing scales infinitely otherwise Google could run its search engine in a single server.
Of course an application with severe memory leakage will not scale, but the simple fact that your application does not scale well does not mean that it is caused by a memory leak. Also, lets assume that there is indeed a memory leak. You still have to prove that it occurs inside RTL/VCL code and not inside your own code. Sorry, but this is how things work. If you create a new application and it starts leaking memory, the first thing you must do is to double (triple) check your own code and make sure that the leak does not come from your code. Then, try to create a simple test case without external dependencies (so EMBT guys can reproduce the leak using your test case). If the problem is in the ISAPI infrastructure, you will be able to reproduce it for sure. I think this should be very clear.

I'm part of Atozed's IntraWeb development team and I've seen lots of ISAPI IntraWeb applications handling 3000+ simultaneous users. If there was a severe memory leak in ISAPI implementation I think this would be clearly visible in that scenario... Last year I took part in a development project using Delphi + IntraWeb (ISAPI) and it also handles 3000+ users. This specific application connects to a DataSnap server (not the new DataSnap, but the old COM+ based implementation), and by coincidence also uses ADO as the DB access layer.

As I said in my first response: If there is a memory leak in your application, does not matter how many users you have. You should be able to track it down using FastMM in full debug mode. You don't have to debug the ISAPI application (which is only slightly more difficult than debugging a desktop app, BTW). You just need to run your ISAPI application using FastMM in full debug mode. Again, you don't need thousands of users to make a memory leak visible. A single user should also reveal it.

Believe me, sometimes we can swear that our application is 100% perfect, 100% of the times, but things like FastMM in full debug mode may reveal very interesting things.
Brian Gochnauer

Posts: 6
Registered: 10/26/00
Re: ISAPI DLL memory leaks  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 22, 2016 8:44 AM   in response to: Alexandre Machado in response to: Alexandre Machado
First, scalability has nothing to do with memory leaks necessarily. Even applications with zero memory leaks may not scale well. Also, nothing scales infinitely otherwise Google could run its search engine in a single server.
Of course an application with severe memory leakage will not scale, but the simple fact that your application does not scale well does not mean that it is caused by a memory leak. Also, lets assume that there is indeed a memory leak. You still have to prove that it occurs inside RTL/VCL code and not inside your own code. Sorry, but this is how things work. If you create a new application and it starts leaking memory, the first thing you must do is to double (triple) check your own code and make sure that the leak does not come from your code. Then, try to create a simple test case without external dependencies (so EMBT guys can reproduce the leak using your test case). If the problem is in the ISAPI infrastructure, you will be able to reproduce it for sure. I think this should be very clear.

I'm part of Atozed's IntraWeb development team and I've seen lots of ISAPI IntraWeb applications handling 3000+ simultaneous users. If there was a severe memory leak in ISAPI implementation I think this would be clearly visible in that scenario... Last year I took part in a development project using Delphi + IntraWeb (ISAPI) and it also handles 3000+ users. This specific application connects to a DataSnap server (not the new DataSnap, but the old COM+ based implementation), and by coincidence also uses ADO as the DB access layer.

As I said in my first response: If there is a memory leak in your application, does not matter how many users you have. You should be able to track it down using FastMM in full debug mode. You don't have to debug the ISAPI application (which is only slightly more difficult than debugging a desktop app, BTW). You just need to run your ISAPI application using FastMM in full debug mode. Again, you don't need thousands of users to make a memory leak visible. A single user should also reveal it.

Believe me, sometimes we can swear that our application is 100% perfect, 100% of the times, but things like FastMM in full debug mode may reveal very interesting things.

A bit harsh, although I may be ignorant; I don't believe I'm not an idiot.
Not much public information on ISAPI dll design with Delphi, nearly always shown using standalone GUI interface.

I was asking; in a round-about way i guess; if there are certain things in the design of the ISAPI dll using ADO components to SQL Server that generally don't scale well or 'not a good idea' because of resources that can't be recovered (memory leaks).

The application; at peak; runs more than 3,000 transactions a second and the database is not yet running at 100%. So it scales to my needs.

But back to my ignorance; I cannot seem to get FastMM4 to run and report to a file outside of the IDE debugger. If you or anyone else has any wisdom in that regard.

compiled with debug; then run outside IDE results in no log but a popup saying there was a leak

 program SNAP2;
{$APPTYPE GUI}
{$IFDEF DEBUG}
  {$INCLUDE FastMM4Options.inc}
  {$DEFINE FullDebugMode}
  {$Define LogMemoryLeakDetailToFile}
  {$DEFINE EnableMemoryLeakReporting}
{$ENDIF DEBUG}
 
uses
  {$IFDEF DEBUG} Fastmm4,{$ENDIF DEBUG}
  Vcl.Forms,
  Web.WebReq,
  IdHTTPWebBrokerBridge,
  FormUnit2 in 'FormUnit2.pas' {Form1},
  ServerContainerUnit2 in 'ServerContainerUnit2.pas' {ServerContainer2: TDataModule},
  WebModuleUnit2 in 'WebModuleUnit2.pas' {WebModule2: TWebModule},
  ServerMethodsUnit2 in 'ServerMethodsUnit2.pas' {ServerMethods2: TDSServerModule};
 
{$R *.res}
 
begin
  {$IFDEF DEBUG}
    ReportMemoryLeaksOnShutdown := True;
    SetMMLogFileName('C:\temp\snap2.log');
  {$ENDIF DEBUG}
 
  if WebRequestHandler <> nil then   WebRequestHandler.WebModuleClass := WebModuleClass;
  Application.Initialize;
  Application.CreateForm(TForm1, Form1);
  Application.Run;
end.


Thanks,
Brian

Edited by: Brian Gochnauer on Jan 22, 2016 9:05 AM
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02