- To determine if the import is a message catalog upgrade the package - version of the catalog file will be compared with the highest package version - currently in the system (in the database). If the package version in the - catalog file deviates from what is in the system then the import is considered an upgrade. + The import should be considered an upgrade if the package has had messages + imported before. In this case the proc lang::catalog::import_messages will be used + to register the new messages with the system and handle the upgrade logic (a merge + with what's in the database).
-
- For upgrades, changed messages will have their old values (the ones in the
- database that are overwritten) backed up to a file with a name on a format along the lines of
- overwritten_messages_upgrade.
+ Import a given set of messages from a catalog file to the database + for a certain package and locale. If we already have messages in the db + for the given package and locale then a merge + between the database messages and the file messages will be performed. +
+ ++ Foreach message to import, the base + messages for the merge is the messages in the db from the last time + db and catalog file were in sync for the corresponding message key. The first such sync point + is the initial import of a message. After that, any export of messages to + the file system will be a sync point. Also, after an upgrade, a large number + of the resulting messages in the db will be identical to those in the file (the + file messages take precedence on conflict) and those messages will also be sync points. + A message being in sync between db and file is indicated by the lang_message.sync_time + column being set to a not null value. +
+ ++ This proc is idempotent which means that it can be executed multiple times and after the first + time it's been executed it won't have any effect on the db. See the corresponding + acs-automated-testing test case called upgrade. +
+ ++ What follows below is a description of the logic of the proc in terms of its input, the cases considered, + and the logical actions taken for each case. +
+ ++ There are three sets of keys, file, db, and base keys. For each key in + the union of these keys there are three messages that can exist: the file message, the db message, and the base message. The + base message serves as the base for the merge. We will distinguish all the different permutations + of each of the three messages existing or not, and all permutations of the messages being different from eachother. + We don't distinguish how two messages are different, only whether they are different or not. + In total that gives us 14 cases (permutations) to consider. +
+ ++ *** Exactly one of messages exists (3 cases): + + 1. base message (deleted in file and db). upgrade_action=none, conflict_p=f + + 2. db message (added in db). upgrade_action=none, conflict_p=f + + 3. file message (added in file). upgrade_action=add, conflict_p=f + + *** Exactly two of the messages exist (6 cases): + + - Base and file message (db message deleted): + 4. Differ (conflicting change). upgrade_action=resurrect, conflict_p=t + 5. No difference (no conflicting change). upgrade_action=none, conflict_p=f + + - Base and db message (file message deleted): + 6. Differ (conflicting change): upgrade_action=delete, conflict_p=t + 7. No difference (no conflicting change): upgrade_action=delete, conflict_p=f + + - File and db message (message added in both db and file): + 8. Differ (conflicting change). upgrade_action=update, conflict_p=t + 9. No difference (identical changes). upgrade_action=none, conflict_p=f + + *** All three messages exist (5 cases): + + 10. All the same. upgrade_action=none, conflict_p=f + + 11. File and base the same. upgrade_action=none, conflict_p=f + + 12. DB and base the same. upgrade_action=update, conflict_p=f + + 13. File and DB the same. upgrade_action=none, conflict_p=f + + 14. All different. upgrade_action=update, conflict_p=t ++ + @param file_messages_list An array list with message keys as keys and + the message of those keys as values, + i.e. (key, value, key, value, ...) + + @param package_key The package_key for the messages. + + @param locale The locale of the messages. + + @return An array list containing the number of messages processed, number of messages added, + number of messages updated, and the number of messages deleted by the import. The keys of the + array list are processed, added, updated, and deleted. + + @author Peter Marklund + @author Lars Pind +} { + set message_count(processed) 0 + set message_count(added) 0 + set message_count(updated) 0 + set message_count(deleted) 0 + + # Form arrays for all three sets of messages + array set file_messages $file_messages_list + array set db_messages [messages_in_db \ + -package_key $package_key \ + -locale $locale] + array set base_messages [last_sync_messages \ + -package_key $package_key \ + -locale $locale] + + foreach arrname { base_messages file_messages db_messages } { + set dummy [list] + foreach elm [lsort [array names $arrname]] { + lappend dummy "$elm=[set ${arrname}($elm)]" + } + ns_log Debug "lang::catalog::import_messages - $arrname: $dummy" + } + + # Remember each time we've processed a key, so we don't process it twice + array set message_key_processed_p [list] + + # Loop over the union of import and db keys. + foreach message_key [lsort [concat [array names db_messages] [array names file_messages] [array names base_messages]]] { + if { [info exists message_key_processed_p($message_key)] } { + continue + } + set message_key_processed_p($message_key) 1 + + ########################################### + # + # Figure out how db and file messages have changed with regards to the base message + # + ########################################### + + # The variables indicate how the db and file messages have changed + # from the base message. Valid values are: none, add, update, delete + set db_change "none" + set file_change "none" + + if { [info exists base_messages($message_key)] } { + # The base message exists + + if { [info exists db_messages($message_key)] } { + # db message exists + if { ![string equal $db_messages($message_key) $base_messages($message_key)] } { + # db message and base message differ + set db_change "update" } + } else { + # db message does not exist + set db_change "delete" } + + if { [info exists file_messages($message_key)] } { + # file message exists + if { ![string equal $file_messages($message_key) $base_messages($message_key)] } { + # file message and base message differ + set file_change "update" + } + } else { + # file message does not exist + set file_change "delete" + } + } else { + # The base message does not exist + + if { [info exists db_messages($message_key)] } { + # db message exists + set db_change "add" + } + if { [info exists file_messages($message_key)] } { + # file message exists + set file_change "add" + } } - } - # Loop over and register the messages - array set overwritten_db_messages {} - foreach message_key $messages_array_names { - set qualified_key "$package_key.$message_key" - set new_message $messages_array($message_key) + ########################################### + # + # Based on the change in file and db messages, + # and based on whether file and db messages differ, decide + # which upgrade actions to take + # + ########################################### - # Failing to register one message should not cause the whole file import to fail - with_catch errmsg { - # If this is an upgrade - save old message if it will be overwritten - if { $upgrade_p } { - # Check if the message existed previously - if { [lang::message::message_exists_p $locale $qualified_key] } { - # Check if message is updated, avoid variable substitution during lookup by setting upvar_level to 0 - set old_message [lang::message::lookup $locale $qualified_key {} {} 0] - if { ![string equal $old_message $new_message] } { - set overwritten_db_messages($message_key) $old_message + # Default values cover the cases 2, 5, 9, 10, 11, 13 + set import_case "in 2, 5, 9, 10, 11, 13" + set upgrade_status "no_upgrade" + set conflict_p "f" + + switch $db_change { + none { + switch $file_change { + none {} + add { + # case 3 + set import_case 3 + # add message from file to db + set upgrade_status "added" } + update { + # case 12 + set import_case 12 + # update db with file message + set upgrade_status "updated" + } + delete { + # case 7 + set import_case 7 + # mark message in db deleted + set upgrade_status "deleted" + } } - } + } + add { + switch $file_change { + none {} + add { + if { ![string equal $db_messages($message_key) $file_messages($message_key)] } { + # case 8 + set import_case 8 + # differing additions in db and file + set upgrade_status "updated" + set conflict_p "t" + } + } + } + } + update { + switch $file_change { + none {} + update { + if { ![string equal $db_messages($message_key) $file_messages($message_key)] } { + # case 14 + set import_case 14 + # differing updates in file and db + set upgrade_status "updated" + set conflict_p "t" + } + } + delete { + # case 6 + set import_case 6 + # deletion in file but update in db + set upgrade_status "deleted" + set conflict_p "t" + } + } + } + delete { + switch $file_change { + none {} + update { + # case 4 + set import_case 4 + # deletion in db but update in file + set upgrade_status "added" ;# resurrect + set conflict_p "t" + } + delete { + # case 1 + set import_case 1 + # deletion in both db and file + # no status change, no conflict + # sync time should be updated below + } + } + } + } - # Register the new message with the system + ########################################### + # + # Execute upgrade actions + # + ########################################### + + # For certain messages we need to move the sync point so that we have a current base for the next upgrade. + if { [string equal $db_change "none"] || ![string equal $file_change "none"] } { + # If there is no db change then any change in the file will be reflected in + # db (file takes precedence) and file and db are identical. + # Also, regardless of what's happened in db, if + # there has been a change in the file then that change will take effect in + # the db and file and db are again identical (in sync). + set update_sync_p 1 + } else { + set update_sync_p 0 + } + + # Store a new message in the database if we are adding or updating + if { [string equal $upgrade_status "added"] || [string equal $upgrade_status "updated"] } { + + ns_log Debug "lang::catalog::import_messages - invoking lang::message::register with import_case=\"$import_case\" -update_sync=$update_sync_p $message_key $upgrade_status $conflict_p" lang::message::register \ - -upgrade=$upgrade_p \ - $catalog_array(locale) \ - $catalog_array(package_key) \ - $message_key \ - $new_message + -update_sync \ + -upgrade_status $upgrade_status \ + -conflict=$conflict_p \ + $locale \ + $package_key \ + $message_key \ + $file_messages($message_key) - if { [info exists descriptions_array($message_key)] } { - lang::message::update_description \ - -package_key $catalog_array(package_key) \ - -message_key $message_key \ - -description $descriptions_array($message_key) - } - } { - global errorInfo - ns_log Error "Registering message for key $qualified_key in locale $locale failed with error message \"$errmsg\"\n\n$errorInfo" + } elseif { $update_sync_p || [string equal $upgrade_status "deleted"] } { + # Set the upgrade_status, deleted_p, conflict_p, and sync_time properties of the message + + # If we are doing nothing, the only property of the message we might want to update in the db + # is the sync_time as we might have discovered that db and file are in sync + array unset edit_array + if { ![string equal $upgrade_status "no_upgrade"] } { + set edit_array(upgrade_status) $upgrade_status + set edit_array(deleted_p) [string equal $upgrade_status "deleted"] + set edit_array(conflict_p) $conflict_p + } + + ns_log Debug "lang::catalog::import_messages - invoking lang::message::edit with import_case=\"$import_case\" -update_sync=$update_sync_p $message_key [array get edit_array]" + lang::message::edit \ + -update_sync=$update_sync_p \ + $package_key \ + $message_key \ + $locale \ + [array get edit_array] + } else { + ns_log Debug "lang::catalog::import_messages - not doing anything: import_case=\"$import_case\" $message_key $upgrade_status $conflict_p" } - } - # Save any messages overwritten in database - if { $upgrade_p && [array size overwritten_db_messages] > 0 } { - set system_package_version [system_package_version_name $package_key] - # Note that export_messages_to_file demands a certain filename format + if { [lsearch -exact {added updated deleted} $upgrade_status] != -1 } { + incr message_count($upgrade_status) + } else { + incr message_count(processed) + } - ns_log Notice "lang::catalog::import_from_file: Saving overwritten messages during upgrade for package $package_key and locale $locale in file $filename" - set file_path [get_catalog_file_path \ - -backup_from_version ${system_package_version} \ - -backup_to_version $catalog_array(package_version) \ - -package_key $package_key \ - -locale $locale] - export_messages_to_file $file_path [array get overwritten_db_messages] - } + } ;# End of message key loop + + return [array get message_count] } ad_proc -public lang::catalog::import { @@ -657,23 +966,35 @@ } { Import messages from catalog files to the database. By default all messages for enabled packages and enabled locales will be imported. Optionally, the import - can be restricted to a certain package and/or a list of locales. + can be restricted to a certain package and/or a list of locales. Invokes the proc + lang::catalog::import_messages that deals with multiple imports (upgrades). @param package_key Restrict the import to the package with this key @param locales A list of locales to restrict the import to @param initialize Only load messages from packages that have never before had any message imported @param cache Provide this switch if you want the proc to cache all the imported messages + @return An array list containing the number of messages processed, number of messages added, + number of messages updated, and the number of messages deleted by the import. The keys of the + array list are processed, added, updated, and deleted. + + @see lang::catalog::import_messages + @author Peter Marklund } { + set message_count(processed) 0 + set message_count(added) 0 + set message_count(updated) 0 + set message_count(deleted) 0 + if { ![empty_string_p $package_key] } { set package_key_list $package_key } else { set package_key_list [apm_enabled_packages] } if { $initialize_p } { - set uninitialized_packages [db_list select_uninitialized {}] + set uninitialized_packages [uninitialized_packages] } foreach package_key $package_key_list { @@ -693,23 +1014,30 @@ # Issue a warning and exit if there are no catalog files if { [empty_string_p $catalog_files] } { - ns_log Warning "lang::catalog::import: No catalog files found for package $package_key" + ns_log Warning "No catalog files found for package $package_key" continue } + array unset loop_message_count foreach file_path $catalog_files { # Use a catch so that parse failure of one file doesn't cause the import of all files to fail - if { [catch {import_from_file $file_path} errMsg] } { + if { [catch { array set loop_message_count [lang::catalog::import_from_file $file_path] } errMsg] } { global errorInfo - ns_log Error "lang::catalog::import: The import of file $file_path failed, error message is:\n\n${errMsg}\n\nstack trace:\n\n$errorInfo\n\n" - } - } + ns_log Error "The import of file $file_path failed, error message is:\n\n${errMsg}\n\nstack trace:\n\n$errorInfo\n\n" + } + } + + foreach action [array names loop_message_count] { + set message_count($action) [expr $message_count($action) + $loop_message_count($action)] + } } if { $cache_p } { lang::message::cache } + + return [array get message_count] } ad_proc -private lang::catalog::get_catalog_paths_for_import { @@ -756,7 +1084,7 @@ if { [file exists $file_path] } { lappend catalog_files $file_path } else { - ns_log Error "lang::catalog::get_catalog_paths_for_import: Catalog file $file_path not found. Failed to import messages for package $package_key and locale $locale" + ns_log Error "Catalog file $file_path not found. Failed to import messages for package $package_key and locale $locale" } } @@ -817,7 +1145,7 @@ if [catch { set translated_message [lang_babel_translate $message en_$lang] } errmsg] { - ns_log Notice "lang::catalog::translate: Error translating $message into $lang: $errmsg" + ns_log Notice "Error translating $message into $lang: $errmsg" } else { lang::message::register $lang $package_key $message_key $translated_message }