Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: Delphi XE2 and DEP


This question is answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 19 - Last Post: Jun 29, 2016 10:34 AM Last Post By: Stephen Boyd
Stephen Boyd

Posts: 12
Registered: 2/1/98
Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 6, 2016 11:52 AM
I have an windows service written using Delphi XE2, IBObjects and Indy. This program is a simple SMTP mail server. On Windows 2012 R2 machines with DEP enabled only for windows executables it is still failing every few minutes with what appear to be DEP errors. The program simply stops running without trigger any of my try ... except blocks. There is an entry in the event log telling me that the program crashed with error code 0x0c.

I have always had to run my Delphi XE2 apps with DEP disabled and, up until now, that has solved the problem.

But this one has me stumped. Any guidance is appreciated.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 6, 2016 12:17 PM   in response to: Stephen Boyd in response to: Stephen Boyd
Stephen wrote:

I have always had to run my Delphi XE2 apps with DEP disabled
and, up until now, that has solved the problem.

Are you using the {$SetPEOptFlags} directive to enable/disable the DEP flag
in your executable's PE header?

But, either way, if you are getting a DEP violation at runtime then some
piece of code in your app is not playing nice with DEP. It is trying to
execute machine code from a non-execute memory block. You need to track
that down. Look for any pieces of code that allocate a memory block dynamically
and then puts machine code in it, but doesn't mark the memory as PAGE_EXECUTE
with VirtualAlloc() or VirtualProtect().

FYI, there is a known issue with Delphi's delay-load mechanism disabling
ASLR:

Delphi EXE has no ASLR
https://forums.embarcadero.com/thread.jspa?threadID=171604

Maybe there is a another bug that affects DEP? Who knows.

--
Remy Lebeau (TeamB)
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 6, 2016 12:52 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:

Are you using the {$SetPEOptFlags} directive to enable/disable the DEP flag
in your executable's PE header?
No. If I check the process status using process explorer it shows DEP and ASLR disabled.

But, either way, if you are getting a DEP violation at runtime then some
piece of code in your app is not playing nice with DEP. It is trying to
execute machine code from a non-execute memory block. You need to track
that down. Look for any pieces of code that allocate a memory block dynamically
and then puts machine code in it, but doesn't mark the memory as PAGE_EXECUTE
with VirtualAlloc() or VirtualProtect().

Our code certainly doesn't do that for the simple reason that we wouldn't know how. We write relatively simple VCL applications using stock Delphi tools. A search of our source library for VirtualAlloc shows no references. VirtualProtect appears in RWMFullFixUnit.pas, which we need to use to fix the Delphi global atom leak. But I have to confess that I do not understand what this bit of code does at all. I just know that it fixes the global atom leak.

Who knows?

Who knows indeed. That's what I'm trying to find out.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 6, 2016 1:50 PM   in response to: Stephen Boyd in response to: Stephen Boyd
Stephen wrote:

No. If I check the process status using process explorer it shows DEP
and ASLR disabled.

Well, have you tried enabling the DEP flag in your DPR file?

{$SetPEOptFlags $40}


Our code certainly doesn't do that for the simple reason that we
wouldn't know how.

Perhaps, but that doesn't mean that none of your 3rd party code is not doing
it. The VCL does, for instance (in MakeObjectInstance(), which is used by
AllocateHWND(), which every TWinControl uses, and a few others like TTimer),
but at least it does mark the memory as PAGE_EXECUTE. But maybe some other
piece of 3rd party code is not. Hard to say without a code review.

We write relatively simple VCL applications using stock Delphi tools.

IBObjects is not stock. Indy is not stock. They are 3rd party (I can tell
you Indy does not have any known DEP issues, though).

A search of our source library for VirtualAlloc shows no references.
VirtualProtect appears in RWMFullFixUnit.pas, which we need to use
to fix the Delphi global atom leak.

I know that fix (and it is not the only fix available), and it should not
be related to DEP. Although it does patch the PE Imports table, so maybe
DEP is not liking that?

But I have to confess that I do not understand what this bit of code
does at all. I just know that it fixes the global atom leak.

It is hooking the Win32 RegisterWindowMessage() function to detect when the
VCL attempts to dynamically register the RM_GetObjectInstance message ID
using a unique per-process name. There is an old bug in the Controls unit
(see http://qc.embarcadero.com/wc/qcmain.aspx?d=90511, which was fixed in
XE3) where each new instance of the VCL allocates a new message ID and does
not release it (because it cant, there is no API for it). As new VCL instances
are loaded and unloaded over time, the atoms used to back those message IDs
can add up until the global atom table fills up. The "fix" checks if a message
ID with an offending name already has an atom available and if so then just
returns that atom as-is instead of letting the OS allocate a new one. The
proper fix was for the VCL to stop registering RM_GetObjectInstance with
unique per-process names to begin with, thus only 1 global atom is ever allocated
for multiple processes.

--
Remy Lebeau (TeamB)
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 6, 2016 2:02 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:
Stephen wrote:

We write relatively simple VCL applications using stock Delphi tools.

IBObjects is not stock. Indy is not stock. They are 3rd party (I can tell
you Indy does not have any known DEP issues, though).

I'll give you IBObjects, but Indy ships with Delphi. So, while it may not actually be written by Embarcadero, to my mind it is a stock Delphi object.

Still looking at the code ...
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 7, 2016 12:27 PM   in response to: Stephen Boyd in response to: Stephen Boyd
After more analysis of the problem here is what I have found:

In case I failed to mention it before, this program is a Windows service implemented using the Delphi TService component.

I have 5 copies of this service running. Each one for a different company. When one copy of the service fails, they all fail at the same time. This leads me to believe that this is more of a Windows issue that a bug in the program code. At least some Windows event is triggering it. But I could be wrong.

All of the logic for the service is implemented in TService.OnExecute. Which means that in the main processing loop I have the following:
    while (not Termnated) do
    begin
        ...
        ServiceThread.ProcessRequests;
        Sleep(100);
    end;


The program is failing in the Sleep(100) call. I know this because I write a line to the trace file before the Sleep and one after. When the program fails I see the before Sleep line but not the after Sleep line.

The service will run an apparently random length of time before failure. Anywhere from a few seconds to an hour or more.

If this were being caused by some weird stack corruption bug in the program I would expect the individual copies of the service to fail randomly, not all at the same time. That is to say, a failure of one copy should not be causing the failure of all the other copies.

This is totally bizarre and I am at a loss.
Peter Below

Posts: 1,227
Registered: 12/16/99
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 7, 2016 12:51 PM   in response to: Stephen Boyd in response to: Stephen Boyd
Stephen Boyd wrote:

After more analysis of the problem here is what I have found:

In case I failed to mention it before, this program is a Windows
service implemented using the Delphi TService component.

I have 5 copies of this service running. Each one for a different
company. When one copy of the service fails, they all fail at the
same time. This leads me to believe that this is more of a Windows
issue that a bug in the program code. At least some Windows event is
triggering it. But I could be wrong.

All of the logic for the service is implemented in
TService.OnExecute.

Don't do that. Do not handle OnExecute at all. Instead create a worker
thread in the OnStart event and have the thread do all the work
instead. You need to handle OnStop to stop and destroy the thread
again, and may want to implement OnPause and OnResume to control the
thread.


--
Peter Below
TeamB

Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 7, 2016 1:34 PM   in response to: Peter Below in response to: Peter Below
Peter Below wrote:

Don't do that. Do not handle OnExecute at all. Instead create a worker
thread in the OnStart event and have the thread do all the work
instead. You need to handle OnStop to stop and destroy the thread
again, and may want to implement OnPause and OnResume to control the
thread.

OK. I moved all of the logic for the service to a thread that I start in OnStart. It was still giving me the same error in the same place. i.e. the Sleep was bombing. So, I added a TEvent and a TTimer and removed the Sleep. So in myThread.Execute I now have:

    while (not Terminated) do
    begin
        ...
        while ((not Terminated) and (myEvent.WaitFor(100) <> wrSignaled)) do
            ;
    end;


and in the TTimer.OnTimer handler I have:

    myThread.myEvent.SetEvent;


I still have the same problem. Except of course it is not bombing in Sleep it is bombing in myEvent.WaitFor. As far as I can tell.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 7, 2016 3:38 PM   in response to: Stephen Boyd in response to: Stephen Boyd
Stephen wrote:

OK. I moved all of the logic for the service to a thread that I start
in OnStart. It was still giving me the same error in the same place.
i.e. the Sleep was bombing. So, I added a TEvent and a TTimer and
removed the Sleep.

Be careful, TTimer is not thread-safe. You might consider using a waitable
timer instead. Look at the Win32 CreateWaitableTimer() and SetWaitableTimer()
functions. A waitable timer can be used with the WaitFor...() family of
Win32 API functions.

--
Remy Lebeau (TeamB)
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 8, 2016 6:22 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:
Stephen wrote:

OK. I moved all of the logic for the service to a thread that I start
in OnStart. It was still giving me the same error in the same place.
i.e. the Sleep was bombing. So, I added a TEvent and a TTimer and
removed the Sleep.

Be careful, TTimer is not thread-safe. You might consider using a waitable
timer instead. Look at the Win32 CreateWaitableTimer() and SetWaitableTimer()
functions. A waitable timer can be used with the WaitFor...() family of
Win32 API functions.

Didn't know about waitable timers. I will look into it. Not that I expect it to make a difference a this point.
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 8, 2016 8:54 AM   in response to: Stephen Boyd in response to: Stephen Boyd
Stephen Boyd wrote:

Didn't know about waitable timers. I will look into it. Not that I expect it to make a difference a this point.

Replaced the TTimer and TEvent with a waitable timer. No difference.

I tried analyzing the crash dump and from the little my limited experience with windbg could tell, it looks like the stack is corrupt. Top entry on the stack is 0x0 which agrees with the error report which states that the program is trying to reference a zero pointer. The trouble is I have no idea how this is happening. Especially in 5 separate executable images simultaneously.

Just to add to the confusion, I have been running this on my Windows 10 machine all morning with no problems at all. But the copies running on a Windows 2012 R2 server are crashing frequently. Sadly, this isn't the first time I have run into things that run flawlessly everywhere except Windows 2012 R2. Microsoft took a great big hammer to Windows when they made 2012 and seem to have messed quite a few things up. You'll never get them to admit that of course. :(
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 7, 2016 3:36 PM   in response to: Stephen Boyd in response to: Stephen Boyd
Stephen wrote:

I have 5 copies of this service running.

And how are you accomplishing that exactly? TService is not designed for
that, it takes extra work to enable that kind of functionality.

When one copy of the service fails, they all fail at the same time.

That might be an indication of a lower-level failure that is not related
to your actual code at all, your code might just be an innocent victim of
circumstance.

The program is failing in the Sleep(100) call.

Unless memory is being corrupted, there is no way a simple Sleep() can be
causing a DEP violation.

I know this because I write a line to the trace file before the Sleep and
one after.
When the program fails I see the before Sleep line but not the after Sleep
line.

That is actually not a good indication that Sleep() is the actual culprit.
It is possible that the event data is being cached by the OS and the DEP
violation kills the process before the event data gets flushed to the trace
file.

--
Remy Lebeau (TeamB)
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 8, 2016 6:20 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:
Stephen wrote:

I have 5 copies of this service running.

And how are you accomplishing that exactly? TService is not designed for
that, it takes extra work to enable that kind of functionality.

Quite easily. When setting up the service I pass a command line parameter indicating the company ID. In the .DPR for the service I change the service name to include the company ID, making it unique. This is done before the call to Application.Run. So, in the windows services configuration program you see 5 separate services one for each company, each with a unique company ID passed on the command line.

When one copy of the service fails, they all fail at the same time.

That might be an indication of a lower-level failure that is not related
to your actual code at all, your code might just be an innocent victim of
circumstance.

Definitely what I am beginning to think. Is there any way to get a crash dump when program crashes and get a stack trace of some kind. Bearing in mind that the DEP exception is not trappable in the Delphi code.

The program is failing in the Sleep(100) call.

Unless memory is being corrupted, there is no way a simple Sleep() can be
causing a DEP violation.

That was my assumption but how can memory be corrupted in 5 separate executables at the same instant?

I know this because I write a line to the trace file before the Sleep and
one after.
When the program fails I see the before Sleep line but not the after Sleep
line.

That is actually not a good indication that Sleep() is the actual culprit.
It is possible that the event data is being cached by the OS and the DEP
violation kills the process before the event data gets flushed to the trace
file.

Possible. It could also be that the sleep is just yielding control allowing another thread to to something that caused the DEP.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 8, 2016 9:22 AM   in response to: Stephen Boyd in response to: Stephen Boyd
Stephen wrote:

That was my assumption but how can memory be corrupted in
5 separate executables at the same instant?

Maybe the HDD or RAM is suffering from a hardware failure.

--
Remy Lebeau (TeamB)
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 8, 2016 10:44 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:

Maybe the HDD or RAM is suffering from a hardware failure.

Usually when that happens you see entries the the Windows event log. There don't seem to be any. Doesn't mean there isn't a hardware problem though.

To add more to the confusion. The program has a compile time switch that allows me to run the program as a desktop application rather than as a service. I usually only use that when I am debugging the program. Just for grins, I compiled it with that switch set and I am now running one copy of the service on my desktop instead of as a service. The services have gone through several crash / respawn cycles but the one on the desktop is still running. That would seem to indicate that there is nothing wrong with the code since the only difference between the two versions is calling ServiceStart rather than Application.Run in the .DPR.

    {$IFDEF TEST}
        AurEmailService.ServiceStart(nil, started);
        while (True) do
            Sleep(100);
    {$ELSE}
        WriteTrace('Running application ...', []);
        Application.Run;
    {$ENDIF}


Hmmmm!
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 8, 2016 12:55 PM   in response to: Stephen Boyd in response to: Stephen Boyd
Stephen wrote:

The services have gone through several crash / respawn cycles but
the one on the desktop is still running. That would seem to indicate
that there is nothing wrong with the code since the only difference
between the two versions is calling ServiceStart rather than
Application.Run in the .DPR.

There is a BIG difference in the APIs involved when running as an app versus
running as a service. By bypassing Run(), you are bypassing some of the
service APIs being called. But, IIRC, the TService framework does not do
anything weird with memory, so I doubt TService itself is a factor. I still
think something else is going on. Have you tried running the app in real
service mode in the debugger yet? You do know that you can debug a running
service, don't you? You just have to "Attach to Process" once the SCM starts
the service process. Then you can step through the service code like any
other project.

--
Remy Lebeau (TeamB)
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 8, 2016 1:16 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:

There is a BIG difference in the APIs involved when running as an app versus
running as a service. By bypassing Run(), you are bypassing some of the
service APIs being called.

Understood. But it pretty much eliminates my code from the equation. The only TService events that I implement are OnStart, OnStop and OnShutdown. And OnStop and OnShutdown don't enter into it because no one is trying to stop the service.

But, IIRC, the TService framework does not do
anything weird with memory, so I doubt TService itself is a factor.

I have other services, using TService, that run just fine. But they don't use Indy. Not saying Indy is the problem, just saying that is one difference between the services that work and the one that doesn't.

I still think something else is going on.

No doubt.

Have you tried running the app in real
service mode in the debugger yet? You do know that you can debug a running
service, don't you? You just have to "Attach to Process" once the SCM starts
the service process. Then you can step through the service code like any
other project.

Since I can't make it fail on my development system and since the system that it is failing on is 600 miles away behind several firewalls it makes debugging it just a bit difficult. Besides, I'm using Delphi Professional and IIRC it doesn't support remote debugging anyway.

I guess I can rewrite it as a desktop app and just run it in the background using a simple little "service manager" that I have. It allows me to run standard VCL apps (or anything else for that matter) in the background.

Stumped and getting grumpy. :(
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 29, 2016 7:39 AM   in response to: Stephen Boyd in response to: Stephen Boyd
Well, I finally figured out what the fix was.

When you create a service application Delphi creates an invisible window, much like a data module. So, I blithely put all of my Indy and data access components on this data module. This gets created in the main thread of the service. The components were then accessed from my service thread and this appears to have been causing a conflict periodically. I moved all of the components to a new data module that is created in my service thread and the problem is gone.

Edited by: Stephen Boyd on Jun 29, 2016 10:40 AM
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: Delphi XE2 and DEP [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 29, 2016 9:14 AM   in response to: Stephen Boyd in response to: Stephen Boyd
Stephen wrote:

When you create a service application Delphi creates an
invisible window, much like a data module.

Are you referring to the Forms.TApplication window? Yes, a TService application
still creates that window (and services its message loop) even though a service
is a non-visual background process.

Or are you referring to the Form Designer that the IDE shows for a TService
itself? TService is a descendant of TDataModule.

This gets created in the main thread of the service.

TService itself is created in the main thread, yes. But when the service
is running, TService runs in its own worker thread (the TService.ServiceThread
property), which fires all of the SCM-related service events (OnStart, OnStop,
OnShutdown, OnPause, OnContinue, OnExecute).

The components were then accessed from my service thread
and this appears to have been causing a conflict periodically.

I don't know about IBObjects, but there are no conflicts with creating Indy
components in one thread and accessing them in another thread.

Even then, with any component, cross-thread memory accesses should not be
cauing DEP violations. Unless maybe one thread frees memory that another
thread then accesses, and that is triggering DEP (which I don't see how it
could, since it is not DEP's responsibility to keep track of that, unless
the code is trying to run executable instructions from that memory).

I moved all of the components to a new data module that is
created in my service thread and the problem is gone.

OK.

--
Remy Lebeau (TeamB)
Stephen Boyd

Posts: 12
Registered: 2/1/98
Re: Delphi XE2 and DEP [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 29, 2016 10:34 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:
Stephen wrote:

Or are you referring to the Form Designer that the IDE shows for a TService
itself? TService is a descendant of TDataModule.

Yes. The one that is visible in the IDE.


This gets created in the main thread of the service.

TService itself is created in the main thread, yes. But when the service
is running, TService runs in its own worker thread (the TService.ServiceThread
property), which fires all of the SCM-related service events (OnStart, OnStop,
OnShutdown, OnPause, OnContinue, OnExecute).

Understood.


I don't know about IBObjects, but there are no conflicts with creating Indy
components in one thread and accessing them in another thread.

I have never known it to be a problem. IBObjects does create timers for various purposes and it relies on notification messages as well. One of those might be causing issues.

Even then, with any component, cross-thread memory accesses should not be
cauing DEP violations. Unless maybe one thread frees memory that another
thread then accesses, and that is triggering DEP (which I don't see how it
could, since it is not DEP's responsibility to keep track of that, unless
the code is trying to run executable instructions from that memory).

It's a mystery to me too. It only seems to happen on Windows 2012 R2. This is not the first thing that I have run across that works on all Windows versions except 2012 R2.
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02