Comments you submit will be routed for moderation. If you have an account, please log in first.
Modify

Ticket #4222 (new task)

Opened 5 years ago

Last modified 4 years ago

max_neval semantics

Reported by: wehart Owned by: jdsiiro
Priority: major Milestone: Acro 2.1
Component: Colin Version: Colin 2.0
Keywords: Cc: jwatson
Subcomponent: Source Code

Description

I'd like to finalize the semantics for 'maximum number of evaluations' for the COLIN 3.0 release.

In COLIN 2.0, there is a distinction between (1) the total number of allowed evaluations, and (2) the number of evaluations allowed by the current execution of an optimizer.

(2) is probably what most users think of when they think of a termination rule for optimizers like GAs.

(1) is a termination rule that makes sense for simple hybrid methods that share a single optimization application.

I'm not sure that (1) makes sense in the broader context that we're defining for COLIN 3.0. If we don't know who we're sharing an optimization application with, does this termination rule make sense?

I _think_ we should redefine max_neval to only refer to context (2). But, how would we define (1) for solvers like ABO? Certainly, it makes sense to have global-level control of this sort of hybridization management...

Attachments

Change History

comment:1 Changed 5 years ago by wehart

  • Priority changed from normal to blocker

I'm making this priority 'blocker', since this impacts the COLIN 3.0 solver integration.

comment:2 Changed 5 years ago by wehart

There's a related issue with this. COLIN 2.0 solvers test for the number of evaluations by comparing the problem->eval_count() method with the termination limit (for (2)). Is that well-defined when multiple solvers are sharing an application? If not, how do we manage this issue...?

comment:3 Changed 5 years ago by jdsiiro

I'll add one more wrinkle: is max_neval a limit on the total number of evaluations requested, or on the number of times the actual evaluation function was called (i.e. exclude the count of cache hits).

comment:4 Changed 5 years ago by wehart

For most solvers, whether evaluations are due to cache hits is immaterial. However, I agree that it would be nice to support cache-aware solvers! For that, I would recommend a separate termination criteria:

max_number_of_uncached_evaluations

comment:5 Changed 5 years ago by wehart

  • Cc jwatson added

Here's my proposal for clarifying the semantics of 'max neval' in the COLIN optimizers:

. Terminate based on the number of function evaluation requests made by the solver. The solver _cannot_ trust that eval_count() accurately computes this, so either (a) we augment the cache manager to include solver-specific counters, or (b) the solver manages this computation itself.

Name: max_number_of_evaluations, max_neval

. Termination based on the total number of function evaluations. The problem->eval_count() is used to provide a termination rule that is global across all solvers that share an application.

Name: max_problem_evaluations

. Termination based on the total number of new (i.e. uncached) evaluations). (I'm not sure how to query the evaluation manager to get this info.)

Name: max_neval_uncached

(We could also have a max_problem_evaluations_uncached.)

comment:6 Changed 5 years ago by wehart

Here are the revised names that John and I agreed on...

max_neval
global_max_neval

max_neval_uncached
global_max_neval_uncached

comment:7 Changed 5 years ago by jdsiiro

  • Priority changed from blocker to major
  • Milestone changed from Colin 3.0 to Colin 3.1

r6082 changed the meaning of OptSolver_Base::neval() to be the number of evaluations requested through the application. While it does not resolve this ticket, this change brings the Colin 3.0 behavior inline with Colin 2.0 so that we can defer this ticket.

comment:8 Changed 5 years ago by jdsiiro

#2340 marked as a duplicate of this ticket.

comment:9 Changed 4 years ago by jdsiiro

  • Milestone changed from Colin 3.1 to acro-dakota 2.0

comment:10 Changed 4 years ago by jdsiiro

  • Milestone acro-dakota 2.0 deleted

Milestone acro-dakota 2.0 deleted

comment:11 Changed 4 years ago by jdsiiro

  • Milestone set to Acro 2.1
View

Add a comment

Modify Ticket

Change Properties
<Author field>
Action
as new
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.