Thoughts on Versioning SMTK Attribute Resources

The question was raised recently as to how to update an existing Attribute Resource w/r to an updated template specification. Here are some of my thoughts on the topic.

To handle the most general case where attribute producing Definitions have been replaced, refactored, or moved to other Attribute Resources, you would have to write an operator to surgically preform the update. In the case where the version updating process would effect multiple resources, I would definitely consider using a SMTK Project and write the updater to update the entire project.

Parts of an Attribute Resource

An Attribute Resource is composed of the following parts:

  • SMTK Resource Information : This includes Properties assigned to the Resource as well as its UUID. Note - for brevity sake I’m also including the Resource’s active set of categories here even though this information is specific to Attribute Resources.
  • Attribute Resource General Workflow Information : This information includes categories, advance levels, and analysis information
  • Attribute Definition Information : This relates to the Attribute Definitions and Expression Evaluators used by the Resource and is usually initially represented by a set of XML Attribute Template files (.sbt)
  • Attribute Information : This includes the Resource’s Components and represent instantiations of the Resource’s Definitions
  • View Specifications : This includes the Views that are used to create the various GUI panels

What Not To Do

You can not just arbitrarily modify an Attribute Definition once attributes have been created from it. If any modification would change the structure of the attribute, you will have lots of issues including unexpected behavior and crashes.

What Can You Safely Modify w/r Attribute and Item Definitions?

Anything that does not effect the information structure of attributes. This would include:

  • Defintion Labels
  • Item Definition Labels
  • Enumeration Strings for ValueItemDefintions
  • Default Values*
  • Advance Levels**
  • Item Definition’s Optional Information**
  • Categories***
  • Analysis Configurations***

* Modifying the default value will not change Item Values that are using the current default
** Changing these may change the validity of an Attribute
*** Changes these require the Attribute Resource to rebuild some runtime information and may also result in a change in Attribute Validity

The Process of Updating To a New Version

Ground Rules

Let’s assume that we have an existing Attribute Resource that represents Version 1 of a workflow and the designer has created a Version 2 of the workflow’s template. Just to be clear I am not referring to a new version of SMTK’s File format. SMTK takes care of reading older file versions and writing out the latest version.

We then have the following situation:

Notice that three sections are not migrated at all. Workflow, Definition and View information is solely defined by the workflow designer. The migration process is focused on the Resource and Attribute Information sections as shown below:

Simplest Migration

Lets assume the following:

  • The V1 Resource Information is valid for V2
    • Assume that the Property values do not refer to Attributes or Definitions
  • The type-names for non-abstract Attribute Definition are the same between versions and Attributes from a V1 Definition type-name are conceptually the same and those created from the V2 Definition of the same type-name
  • If a V1 Attribute Definition does not exist in V2, then its is considered no longer needed and the corresponding Attributes are not to be transferred
  • Item names within an Attribute have not been changed between Versions though their positions within the Attribute may be different. The same is assumed be true regarding the names of children Items within an Item.
  • If an Item name in V1 does not exists in V2, then the information is considered to be no longer needed in V2.

In the above scenario, the migration procedure would consist of the following steps:

  1. Copy UUID and all Properties from V1 to V2
  2. For each Attribute a1 in V1
    a. If its Definition does not exist in V2 then skip it
    b. Use the Attribute Resource V2’s copyAttribute method* to create Attribute a2

* The current copyAttribute process assumes that the Attributes are coming from identical Attribute Definitions and would need the following modifications:

  • Use of Item names rather than their position
  • Allowing the process to proceed if an Item can not be located

Note that the above scenario would also cover the following cases:

  • New Attribute Definitions
  • Insertion of new Item Definitions
  • Deletion of Attribute Definitions that are no longer needed
  • Rearranging of Items with the same Definition
  • Rearranging of Abstract Attribute Definition Hierarchy

How to generalize the process

The designer could provide a set of update scripts that is associated with the Attribute Template:

  • Migration Script - if the designer wants total control of the migration process. If none is specified it assumes that the following is done:
    • Migrate Resource Info
    • Migrate Attribute Info
  • Resource Migration Script - controls how the Properties and Active Categories get migrated to the upgraded Attribute Resource. If none is specified it assumes to simply copy all of the Properties and Active Categories
  • Attribute Migration Script - controls how the Attributes get migrated, if none is specified it assumes that matching the Attribute’s Definition type-name is sufficient
  • Custom Attribute Definition Migration Scripts - controls how a specific Attribute based on its Definition gets migrated. If none is specified it assumes that matching the Item names is sufficient

Issues regarding Associations

There is a question as to whether the association information can be easily transferred to the updated resource. This depends on the following questions:

Are the Prerequisites and Exclusion Information Compatible?

If the information between the version is not the same, then in which cases will it impact the association information? If the new version’s are more permissive, then what was valid based on the original Definition’s Prerequisites and Exclusions should still be valid. The check is relatively simple in that if the new version contains a subset of the original version there should be no effect. If this not true then in theory the new version will need to revalidate the associations.

What about the Definition’s Association Rules?

If the updated and original Definitions have the same association rules then once again (assume the above perquisites and exclusions are compatible) the Persistent Objects that are associated with the original attribute should be associatable with the updated version, else once again the information would beed to be reevaluated. In this case “Comparable” means the following with respects to the association rule:

  • The Acceptance rules of the original version is a subset of the Acceptance rules of the update version
  • The Rejection rules of the updated version is a subset of the Rejection rules of the original version.

Special Note on Handling Attributes from Definitions that have Prerequisites

If you do need to reevaluate association information, you do need to be careful if the Definition has prerequisites. A slight change in the above method could better handle this situation.

  • Instead of going through all of the attributes, you will need to instead traverse all of the Definitions and then traverse all of their Attributes
  • If Definition (D) has a prerequisite, it has generated attributes and has Persistent Objects associated to it, check to see if its prerequisite Definitions have Attributes in the updated version
  • If the prerequisite Definition’s Attributes have not been migrated, pause D’s Attribute migration and instead migrate its prerequisite Definition first and then resume processing D. Note that the prerequisite Definition could itself have prerequisites.

@johnt @dcthomp @rohith @waj334 @mledesma @aron.helser @Aaron FYI - Comments always welcome!

Minor comments:

Just for completeness, this info also includes prerequisites and exclusions sections

I presume the list is for what can be safely modified.

Some code would probably be required in the case where a category or analysis configuration was deleted.

The actual implementation should probably copy properties after the attributes are migrated, and skip copying properties to attributes that aren’t in the new resource.

Thinking ahead a bit, I presume each script would have arguments for (i) the old-version resource, (ii) the new-version resource, (iii) a logger object, and possibly in the future (iv) an object containing the set issues that need to be resolved manually.

I didn’t see anything about changing the V2 resource UUID once the V1 resource’s attributes had been copied into it (and presumably the V1 resource was then un-managed).

Some care will have to be taken in that case as unmanaging a resource R1 may affect associations held by other resources that reference R1 or its components. Also, our recent discussions about when/how resources are removed from a resource manager come into play since the base Operation class should be performing the actual removal. Will it need to do something else to deal with resource swaps/pivots where R2 assumes the UUID of R1? Certainly this phase of updating a resource should happen while locks on both R1 and R2 are held.

  • Concerning Prerequisites and Exclusions - Absolutely correct
  • Will add the word Safely to the heading
  • In terms of 1. Copy UUID and all Properties - I was referring to the Properties directly on the Resource - I am assuming that Properties on the Attributes would be copied during the Attribute Copy Method
  • Thinking ahead - I agree that the signature should support i - iii (not sure about iv)

I would assume migration is done during the loading process so that the original V1 version would not need to be unmanaged since it’s temporary existence would not be observed by anyone.

Proposal: Resource Templates

@Bob_Obara @johnt I have a cunning plan… what if we use resource migration (this process but for any resource type, not just attribute resources) to deal with resource schemas (a.k.a., concepts)? This would let us use modelbuilder itself to author workflow templates.

Similar to many document-based programs (Keynote, Powerpoint, Google Slides), we could provide a “Use this resource as a template” operation and a “Create a resource from a template” operation. We could even provide a simple “Edit template” operation.

The “Use this resource as a template” operation would

  1. Set the “schema” string for the resource to whatever the user specified (bumping the version number as needed if the schema already exists).
  2. Save the resource to a special template directory (e.g., ~/.config/smtk/templates).

The “Create a resource from a template” operation would

  1. Let the user pick a schema from the set in the user’s directory plus any from the modelbuilder installation directory.
  2. Create a new (blank) resource of the proper type and use the migration procedure above to copy the template into the new resource.

The “Edit template” operation would do the same work as “Create a resource from a template” as above, plus

  1. Mark the newly-created resource with an incremented template version number.
  2. Set the newly-created resource’s location to a filename in the special template directory so that if the user saves the resource it will automatically be detected as a template in the future and available for resources based on older versions as a migration target.

Projects like ACE3P or Truchas could bundle resource “templates” in their packages. This could work with projects as well as resources (since projects are resources).