Einstein as limited execution project tries too run too many work units

Darrell
Darrell
Joined: 3 Apr 05
Posts: 12
Credit: 394432205
RAC: 230351
Topic 220067

I have Einstein as a low share (0.33%) and execution limit (Project Max of 5) on my Threadripper 1950X.  It tries to execute on ALL the open CPU threads instead of the limit of the project.

So now I have 20 "Waiting to run" Einstein work units since my other projects had temporarily dried up, but now have sent me more work.  My other projects are also limited to specific numbers, and honor them.

Has this been reported before now?

 

archae86
archae86
Joined: 6 Dec 05
Posts: 3146
Credit: 7088554931
RAC: 1327190

"The Einstein Project" does

"The Einstein Project" does not decide what tasks run on your machine.  Boinc running on your machine decides to launch tasks. (which project, which flavor within project, how many...)

What mechanism are you using to specify a Project Max limit?

Darrell
Darrell
Joined: 3 Apr 05
Posts: 12
Credit: 394432205
RAC: 230351

@ archae86 There seems to be

@ archae86

There seems to be more to this than I first noted.

I am using "app_config.xml" to limit the number of threads of each project to run simultaneously.  All projects honor their respective limits.

The parameter used is <project_max_concurrent>5</project_max_concurrent> and this is working OK in that only 5 threads are actually RUNNING.

I will do more digging to uncover the scenario that caused every open thread to grab an Einstein WU.  Something to do with an empty work queue on other projects perhaps?  I don't yet know.

It is not always clear to me which actor does what in BOINC.  Sometimes it is the Manager, sometimes the Client, so I started here.  You are probably correct in that the Manager is the one doing this.

 

 

Gary Roberts
Gary Roberts
Moderator
Joined: 9 Feb 05
Posts: 5851
Credit: 110739128533
RAC: 32514038

Darrell wrote:It is not

Darrell wrote:
It is not always clear to me which actor does what in BOINC.  Sometimes it is the Manager, sometimes the Client, so I started here.  You are probably correct in that the Manager is the one doing this.

Archae86 didn't blame the manager.  He mentioned BOINC which is the BOINC client.  It's not likely to be anything to do with the manager.

Think of it this way.  If you completely removed BOINC Manager, the BOINC client would be perfectly capable of carrying out all normal operations without a problem.  Those things would include choosing which tasks to run and when, uploading completed results, suspending and restarting tasks from different projects so as to best meet your resource share preferences, checking preferences on the website for any user supplied changes, deciding when and arranging for the downloading of new work, etc, etc.  The manager is not needed for any of this.

The manager is the point and click mechanism for the curious human to spy on what the client is doing - and interfere with it, and adversely affect it if not careful :-)  It's also a mechanism for making some local preference changes which will override the global preferences stored on project websites.  It can be quite complicated so you need to consider and understand the consequences of things you change locally.  Basically, the manager just allows a human to interfere with and perhaps override the sequence of operations the client has decided to perform.

On top of all that, there is the app_config.xml tool for advanced manipulation of individual applications.  It's nothing to do with the manager.  It's all to do with fine-grained control of applications and useful for modifying fractional CPU/GPU budgeting values that may help the client to control the running of tasks more efficiently.  Once again, when you set values this way, you really need to understand the consequences.

As an example, consider the <project_max_concurrent> setting you mention.  You run multiple projects and in your standard website settings, you will already have set your desired resource shares.  BOINC will already be attempting to honour those resource shares.  So if the settings for max_concurrent aren't carefully selected, bearing in mind all the complexities of certain projects having or not having available work at any particular time, you may actually be severely limiting the client from performing its function in an efficient and orderly manner.

What would you expect the client to do if there are unused threads because of lack of available work from more preferred projects but a non-preferred project does have available work ready to go?  I would hope the client would ignore the limit and use the extra threads until such time that a preferred project can supply.  This would appear to be what you have described.  As soon as there are preferred tasks available, the client will suspend the extras and run the preferred tasks.

In other words, what's the point of arbitrarily restricting the client from making the most efficient use of the resources?  If a high resource share project is known to have work shortages, the client will correct for this in the longer term and favour that project when it does have work.  By setting max_concurrent you are probably making it harder for the client to do its job.  The max_concurrent setting is optional.  Unless you really need it for some very specific purpose, it would probably be more efficient to not hamstring the client with that added restriction.

Can you explain the reason why you wish to use this restriction?  Have you really tried giving the client enough time to settle into a pattern which does meet your resource shares in the longer term?  Have you made the client's job more difficult by having too large a work cache setting?  The real danger with that is that if higher resource share projects can't supply work, the client will 'fill up' with lower resource share tasks.  This will come back to haunt you later when the deadline for those extra tasks approaches.  The best chance for the client to work well will occur if you keep your work cache setting even lower with any increase in the number of projects you wish to support.

Cheers,
Gary.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.