1.1 Zero Downtime Migration 21.4 Release Notes
These release notes provide downloading instructions for the latest product software and documentation, and describe new features, fixed bugs, known issues, and troubleshooting information for Zero Downtime Migration Release 21c (21.4).
- What's New in This Release
- Bugs Fixed
- Downloading the Zero Downtime Migration Installation Software
- Downloading the Zero Downtime Migration Documentation
- General Information
- Known Issues
- Troubleshooting
- Additional Information for Migrating to Oracle Exadata Database Service
- Additional Information for Migrating to Oracle Exadata Database Service on Cloud@Customer
- Documentation Accessibility
1.2 What's New in This Release
Zero Downtime Migration Release 21.4 improves the existing 21c functionality with the following enhancements.
-
Physical migration job can pause for Redo Apply to catch up
In a physical migration job, a new response file parameter,
ZDM_APPLY_LAG_MONITORING_INTERVAL
, allows Zero Downtime Migration to monitor redo apply lag after standby database creation. This new phase queries the current redo apply lag, and waits until redo apply is finished before resuming the job.See ZDM_APPLY_LAG_MONITORING_INTERVAL.
-
Physical migration job can upgrade Target time zone file
In a physical migrations job, a new response file parameter,
ZDM_TGT_UPGRADE_TIMEZONE
, allows you to add a post-migration task which upgrades the target database time zone file. This upgrade reverts the target database time zone file to the pre-migration time zone file version that was in the original target database.See ZDM_TGT_UPGRADE_TIMEZONE_FILE.
-
Ignore checks option in ZDMCLI MIGRATE DATABASE command enhanced
An enhancement to the
ZDMCLI migrate database
command-ignore
option allows you to pass a comma-separated list of ignore checks.A new check,
VAULT_CHECK
, has been added to the types of checks that can be ignored.See migrate database.
-
Logical migration GOLDENGATESETTINGS_REPLICAT_* parameters simplified
A new logical migration response file parameter,
GOLDENGATESETTINGS_REPLICAT_PERFORMANCEPROFILE
, simplifies Oracle GoldenGate Replicat configuration. This parameter replaces the need to configure four separateGOLDENGATESETTINGS_REPLICAT_*
parameters listed below.Starting with Oracle Zero Downtime Migration 21c (21.4) release, the following parameters are deprecated and will be desupported in a future release:
GOLDENGATESETTINGS_REPLICAT_MAPPARALLELISM
GOLDENGATESETTINGS_REPLICAT_APPLYPARALLELISM
GOLDENGATESETTINGS_REPLICAT_MAXAPPLYPARALLELISM
GOLDENGATESETTINGS_REPLICAT_MINAPPLYPARALLELISM
If you configure the
GOLDENGATESETTINGS_REPLICAT_PERFORMANCEPROFILE
parameter, then you do not have to configure these deprecated parameters.See GOLDENGATESETTINGS_REPLICAT_PERFORMANCEPROFILE.
-
Logical migration response file parameter for source database administrator wallet path
A new logical migration response file parameter,
WALLET_SOURCEADMIN
, specifies the path to the directory that contains the auto login wallet filecwallet.sso
, which provides the source database administrator password.See WALLET_SOURCEADMIN.
-
CPAT terminology updates adopted
CPAT terminology for pre-migration analysis reports is updated as follows:
- "PASS" with "Passed"
- "INFORMATIONAL" with "Review Suggested"
- "WARNING" with "Review Required"
- "BLOCKER" with "Action Required"
- "FATAL" with "Failed"
Note that earlier Zero Downtime Migration releases allow you to upgrade CPAT, but do not support this new CPAT terminology.
You can see updated example output in Zero Downtime Migration Schema Analysis with Cloud Premigration Advisor Tool
-
Support for tables with XML types stored as CLOBS
Zero Downtime Migration now leverages the Oracle Data Pump CLOB to BINARY STORAGE conversion for XML types.
The logical migration response file parameter,
DATAPUMPSETTINGS_METADATATRANSFORMS
, is enhanced to let you setXMLTYPE_STORAGE_CLAUSE
to 'BINARY XML'.DATAPUMPSETTINGS_METADATATRANSFORMS-1=name:XMLTYPE_STORAGE_CLAUSE, value:'BINARY XML'
With this enhancement all XML types can now be converted as part of the migration.
See DATAPUMPSETTINGS_METADATATRANSFORMS-LIST_ELEMENT_NUMBER
-
Logical migration access to CPAT logs simplified
A new logical migration response file parameter,
COPYCPATREPORTTOZDMHOST
, lets Zero Downtime Migration automatically copy the CPAT report log from the source database server to the ZDM server host job log directory for easier access to logs and more efficient troubleshooting.See COPYCPATREPORTTOZDMHOST
-
DB_NK_CACHE_SIZE values handling enhanced
Zero Downtime Migration now automates the copy of these values from the source init file to the target database. This automation benefits migrations where the source database uses any tablespace of variable block size other than db_block_size. In such cases, without Zero Downtime Migration handling of
DB_NK_CACHE_SIZE
, the restore process would fail.Note that for migrations where non-CDB to PDB conversion takes place, you must set this value in the target placeholder database.
See "Handling of
DB_nK_CACHE_SIZE
" in Source Database Prerequisites -
OCI Object Storage Pre-Authenticated URL supported
Zero Downtime Migration now supports use of OCI Object Storage Pre-Authenticated URL (PAR) to upload Data Pump dump files to an Object Storage bucket. Zero Downtime Migration also supports the PAR URL to download and import Data Pump dump files from the bucket to the target database.
See "Using the Oracle Cloud Object Storage Transfer Medium" in Configuring the Transfer Medium and Specifying Transfer Nodes
-
NoSudo migration
A new authentication plugin,
dbuser
, allows you to perform migrations without sudo access. You can now perform a migration as "oracle" user for all actions on the source and target database hosts for certain migration paths.See Migrating with Database User Privileges
-
METADATA and DATA can be migrated in separate phases
Zero Downtime Migration lets you migrate METADATA and DATA as separate phases.
Workflow performs regular EXPORT and multi-phased import that involves Pre-data Metadata Import to handle USER and PROFILE creation, followed by phase handling METADATA creation and finally the DATA import. This helps with inserting workflow customization in the workflow after user or metadata is created.
See Migrating Metadata
-
Specify the Data Pump job export version
A new logical migration parameter,
DATAPUMPSETTINGS_EXPORTVERSION
, allows you to specify the Data Pump job export version.For example, when migrating a legacy database upgraded from before Oracle 11g (11.2.0.4) and including 11.2.0.4, and that has Workspace manager, it needs the export version "12" to be set in Data Pump Export
VERSION
parameter.See DATAPUMPSETTINGS_EXPORTVERSION
-
Automatic Tablespace Creation
Zero Downtime Migration allows auto creation of user DATA, TEMP, and UNDO tablespaces.
See TABLESPACEDETAILS_AUTOCREATE and Automatic Tablespace Creation
-
Ignore errors on job resumption
In a logical migration job, upon Data Pump job completion, Zero Downtime Migration analyzes the export/import log and identifies the non-ignorable ORA- errors and reports them to the user in the ZDM job log. You can review and ignore these additional export or import errors and proceed to next phase, skipping the export or import operation using new option in
ZDMCLI RESUME JOB
.See Resume a Migration Job
If there are additional Data Pump errors that you want to ignore by default, those can be specified using the parameters IGNOREEXPORTERRORS and IGNOREIMPORTERRORS.
-
Logical Migration without SSH Access
You can perform logical migration to the target database server host without requiring SSH credentials. See Logical Migration without SSH Access.
-
Migrating to Co-Managed Database Server with NFS Data Transfer Medium
This logical migration scenario is supported when the requirements are met in Migrating to Co-Managed Database Server with NFS Data Transfer Medium
-
Resume capability after manual Data Guard switchover
In case of failure during the switchover phase, you can fix any issues and perform a manual switchover. In order to avoid any issues with the migration workflow, this new functionality allows you to specify that a manual switchover has been performed and that Zero Downtime Migration should skip the switchover phase. This can be achieved by resuming the migration job with:
zdmcli resume job -jobid <jobid> -skip SWITCHOVER
See resume job
-
Migrate source databases without SPFILE
Physical Offline migration now supports migrating source databes without SPFILE.
For online migration, Zero Downtime Migration still requires an SPFILE.
See Preparing the Source and Target Databases
-
Physical migration supports TDE wallet specified using WALLET_ROOT
A physical migration can migrate all of the wallet files and sub-directories configured under
WALLET_ROOT
. If the source has PDB wallets in isolated mode, Zero Downtime Migration migrates all of the wallet's files in sub-directories underWALLET_ROOT
. - Disable backup compression and encryption
Zero Downtime Migration allows you to disable backup compression and encryption for physical migration. You can turn off the encryption when migrating between on-premises databases only.
See ZDM_RMAN_ENCRYPT_BACKUP
-
Configurable RMAN section size
You can manually configure the Oracle RMAN section size using the new response file parameter
ZDM_RMAN_SECTION_SIZE
.See ZDM_RMAN_SECTION_SIZE
-
Define PDB target name on non-CDB to PDB conversions
Physical migration allows for conversion between Non-CDB sources and PDB targets, and it is now possible to define the name of the target PDB before starting the migration with the response file parameter
ZDM_NONCDBTOPDB_PDB_NAME
See ZDM_NONCDBTOPDB_PDB_NAME
-
Support for wallet credentials for running a useraction
In a logical migration you can provide an auto login wallet path, which contains the username and its password.
See Registering User Actions and WALLET_USERACTION
-
Pass source export directory name and dump name to a useraction script
You can use the following useraction parameters to pass information to a useraction script
ZDM_DATAPUMP_EXPORT_DIR_NAME=DATA_PUMP_DIR_NAMEZDM_DATAPUMP_EXPORT_DIR_PATH=data_pump_dir_pathZDM_DATAPUMP_DUMP_PREFIX=data_pump_prefix
See Parameters Supplied for Custom Plug-ins with Shell Script User Actions
- Reload objects
You can reload objects and specify the Reload rules. See Selecting Objects for Migration.
-
Migrating Objects With Different Oracle GoldenGate Support Modes
You can migrate objects with different Oracle GoldenGate support modes.
See Migrating Objects With Different Oracle GoldenGate Support Modes
-
Extract and Replicat processes auto start enabled
By default, Zero Downtime Migration ensures that Oracle GoldenGate Extract and Replicat are auto start enabled for resiliency of any unexpected failures for logical migrations.
-
Auto-purge jobs disabled on target
Disable auto-purge jobs on the target database immediately after instantiation (Data Pump Import). Otherwise, the purge jobs will cause data inconsistency condition on the target database causing GoldenGate Replicat to fail. Some of these jobs will purge data. During the ZDM_POST_DATAPUMP_TGT phase, ZDM will disable purge jobs in the target database. ZDM enables the purge jobs in the target database during the ZDM_POST_SWITCHOVER_TGT phase.
-
Logical Migration Best Practices Documentation
Zero Downtime Migration documentation now includes best practice recommendations to help you achieve successful logical migrations.
See Best Practices for Logical Database Migration for more information.
- Installing Zero Downtime Migration on VM in Oracle Cloud Infrastructure
Install ZDM on a VM in OCI by referring to the steps available at Installing Zero Downtime Migration on VM in Oracle Cloud Infrastructure.
- Variables Supplied for Custom Plug-ins with SQL Based User Actions
Information about the SQL based user actions is available at Variables Supplied for Custom Plug-ins with SQL Based User Actions.
- Review for new patches if required by reviewing Patch 33509650: ZDM PATCH USING MOS
Review the latest patch if required by referring to the Install Zero Downtime Migration Software topic.
- Installing Zero Downtime Migration on Red Hat Enterprise Linux 8 in an Oracle Cloud Infrastructure Instance
Refer to the steps available at Installing Zero Downtime Migration on Red Hat Enterprise Linux 8.
1.3 Bugs Fixed
Zero Downtime Migration Release 21.4 introduces the bug fixes listed in the following table.
Table - Bugs Fixed In Zero Downtime Migration Release 21.4
Bug Number | Description |
---|---|
34979586 | LNX-21.3 - ZDM: VALIDATE RDS TO ADB-S MIGRATION FAILS WITH PARAMETER RHPHELPERUTIL-CONSTR-OHOME VALUE IS NOT VALID |
34996189 | ZDM VALIDATION SOURCE FAILS WITH SOURCE AS AIX |
35261857 | ZDM MIGRATION WITH APPLICATION CONTAINER FAILS. |
35325728 | ZDM FAILURE - RMAN-07551: DATA FILE 857 MUST BE RESTORED |
35186823 | PHYSICAL MIGRATION FAILS FOR 11.2.0.4 SOURCES WITH DIRECT DATA TRANSFER -> PRGZ-3635 : ISOLATED MODE KEYSTORE WITHOUT AUTOLOGIN DETECTED FOR PDBS "0" OF DATABASE DBNAME |
35195827 | ZDM: LOGICAL MIGRATION FROM SOLARIS TO EXACC FAILS WITH PRGZ-1141 & PRCZ-4002 ILLEGAL OPTION |
35309075 | ZDM: DBUSER PLUGIN FAILS WITH AN ERROR ON COPY STEP PRCZ-2123 PRCZ-4002 |
35182080 | PRCZ-4002 : FAILED TO EXECUTE COMMAND "/BIN/CP" USING THE PRIVILEGED EXECUTION PLUGIN "DBUSER |
33124731 | ZDM WRONGLY SETS FAL_SERVER ON STANDBY SYSTEM |
33453726 | GG SETUP FOR ID_KEY |
33629992 | ZDM: NONCDB TO PDB CONVERSION FAILS WITH ORA-65122: PLUGGABLE DATABASE GUID CONFLICTS WITH THE GUID OF AN EXISTING CONTAINER |
33647475 | INTERNAL EXCEPTION: JAVA.SQL.SQLNONTRANSIENTCONNECTIONEXCEPTION: TOO MANY CONNECTIONS |
33967375 | ZDM: MIGRATION TO EXACC 12.1 FAILS WITH PRCR-1001 : RESOURCE ORA.ZDM_AUX_SIERRA.DB DOES NOT EXIST |
33991550 | EDZI : ZDM LOGICAL MIGRATION TO ADBD - EVAL OPTION FAILED WITH ERROR PRGZ-1320 : FAILED TO DISCOVER ORACLE DATABASE {0} OBJECTS |
34002538 | ZDM MIGRATION FAILED AT PHASE ZDM_RESTORE_TGT WITH RMAN-11003 ORA-01511 ORA-00261 |
34013080 | MIGRATION -EVAL IGNORING PARAMETER SRC_DB_LISTENER_PORT WHILE EXECUTING PHASE ZDM_PRECHECKS_TGT |
34020210 | ZDM ALLOW NOSUDO DBUSER LOGICAL MIGRATION |
34082591 | SOURCE DB ON OLDER LINUX DISTRIBUTIONS MIGHT FACE ISSUES DUE TO DEPRECATED KEXALGORITHMS |
34082818 | 'CUT' IS NOT LOCATED AT /BIN/CUT IN SEVERAL LINUX DISTRIBUTIONS -? ORADISCOVER.SH FAILS AND SUBSEQUENTLY THE MIGRATION FAILS |
34123897 | ZDM: RESUME ZDM JOB PICKING UP WRONG CASE ORACLE_SID |
34141215 | AS PART OF POST CLOING JOB ON TARGET, SRVCTL RUNS MODIFY COMMAND CHANGING DATABASE NAME TO SAME AS DB UNIQUE NAME |
34209964 | ZDM EVAL - CHARACTER SET PRECHECK |
34241041 | ZDM RESTORE DB PICKS WRONG LOC FOR FIRST ATTEMPT |
34485179 | OCIPROXY_* SETTINGS ARE NOT PASSED TO OCI SDK CAUSES ERROR "JAVA.NET.CONNECTEXCEPTION: CONNECTION REFUSED (CONNECTION REFUSED)" |
34496249 | ZDM: -EVAL FAILS WITH PRCT-1014 : INTERNAL ERROR: RHPHELP_TOOL_ERROR-03 |
34500541 | ZDLRA - DUPLICATE DB FAILING WITH RMAN-06569 WHILE MIGRATION TO EXACC |
34523536 | ZDMLOGICAL: ADD SCHEMA.* TO EXTRACT FOR DDL REPLICATION |
34564746 | ZDMLOGICAL: UPLOAD DUMP FILES FIX IS NEEDED FOR ZDM_UPLOAD_HELPER.SH SCRIPT |
34569153 | ZDMLOGICAL: DBUSER PLUGIN FAILING WITH PRCZ-4001 : FAILED TO EXECUTE COMMAND "/BIN/CHOWN" |
34590652 | ZDM: LOGICAL MIGRATION OF ONE TABLE FAILING WITH ERROR ORA-39038 & ORA-39038 |
34669782 | REVIEW SEGMENT_ATTRIBUTES TRANSFORM FOR ADBCC MIGRATION USING ZDM |
34760462 | BUG FIXES WHILE GETTING ADBCONNECTION DURING VALIDATE_TARGET PHASE |
34860340 | DBUNIQUE_NAME DIFFERENT THAT DB_NAME AT SOURCE REQUIRES MANUAL WORKAROUND |
34893354 | ZDM FAILURE WHEN ZDM JOB DIRECTORY IS HOSTED ON NFS SHARE DUE TO FLOCK() USAGE |
1.4 Downloading the Zero Downtime Migration Installation Software
For a fresh installation of the latest Zero Downtime Migration software version, go to https://www.oracle.com/database/technologies/rac/zdm-downloads.html.
1.5 Downloading the Zero Downtime Migration Documentation
You can browse and download Zero Downtime Migration documentation at https://docs.oracle.com/en/database/oracle/zero-downtime-migration/
1.6 General Information
At the time of this release, there are some details and considerations about Zero Downtime Migration behavior that you should take note of.
1.6.1 Running RHP and Zero Downtime Migration Service on the Same Host
If the Zero Downtime Migration service is installed on the same host where RHP server is deployed, note that there are some workarounds.
If you have has started an RHP server/client on the same node as the Zero Downtime Migration service, using the default port, you must either
-
Stop RHPS/RHPC
-
Modify the port for RHPS/RHPC
This is to avoid port collision between RHP and Zero Downtime Migration. If you don't want to change RHP configuration, you can also modify the port for Zero Downtime Migration before starting the Zero Downtime Migration service.
To identify the ports being used by Zero Downtime Migration:
ZDMinstallation/home/bin/zdmservice status
To stop the Zero Downtime Migration service:
ZDMinstallation/home/bin/zdmservice stop
To modify the ports:
ZDMinstallation/home/bin/zdmservice modify -helpModifies configuration values.USAGE: zdmservice modifyOptional parameters: transferPortRange=<Range_of_ports> rmiPort=<rmi_port> httpPort=<http_port> mysqlPort=<mysql_port>
For example:
ZDMinstallation/home/bin/zdmservice modify mysqlPort=8899Editing MySQL port...Successfully edited port=.* in file my.cnfSuccessfully edited ^\(CONN_DESC=\).* in file rhp.prefSuccessfully edited ^\(MYSQL_PORT=\).* in file rhp.pref
1.6.2 Cloud Premigration Advisor Tool Support
Cloud Premigration Advisor Tool (CPAT) is supported with Zero Downtime Migration for the following use cases:
-
CPAT is run automatically on the source database environment during logical migration jobs from Oracle Cloud and on-premises Oracle Database sources (default behavior)
-
CPAT is run manually from a remote server against an Amazon Web Services RDS Oracle Database source; in other words, CPAT is not run by
ZDMCLI migrate database
(see Running CPAT with a Remote Connection)
CPAT is not supported in the following use cases:
-
Physical migration jobs
-
Generating fixup scripts for Amazon Web Services RDS Oracle Database sources
1.6.3 UNDO Tablespaces Added to the Source Database
Zero Downtime Migration adds UNDO tablespaces to the production database to match the target instance count if the production database has fewer instances.
To prevent Zero Downtime Migration from adding UNDO tablespaces to the source database, you can match the target database nodes count to that of the source database until the switchover, then you can add additional nodes to the target database after the switchover.
1.6.4 Cross-Edition Migration
Zero Downtime Migration cannot be used to migrate an Enterprise Edition database to a Standard Edition database. In the converse case, Standard Edition databases can be migrated to Enterprise Edition databases, but only using the logical migration work flow.
1.6.5 EXT3 File System Support
There is a known issue which prevents Zero Downtime Migration from being installed in EXT3 file systems. The root cause is MySQL bug 102384. This is not a limitation of Zero Downtime Migration, but a constraint from MySQL. When that bug is resolved, Zero Downtime Migration is expected to work on EXT3 file systems.
1.7 Known Issues
At the time of this release, the following are known issues with Zero Downtime Migration that could occur in rare circ*mstances. For each issue, a workaround is provided.
1.7.1 Skip the ZDM RELOAD of empty schema or schema with no qualifying objects
Solution: ZDM filters objects for reload and if there are no objects to be reloaded for any specific schema post applying the following conditions, then avoid the reload feature or do not include the particular schema.
ZDM filters the following objects:
- Objects from
DBA_GOLDENGATE_SUPPORT_MODE
that haveSUPPORT_MODE=NONE
orSUPPORT_MODE=PLSQL
orSUPPORT_MODE= INTERNAL
. - Objects from
DBA_GOLDENGATE_NOT_UNIQUE
that are markedBAD_COLUMN=Y
.ZDM skips
When there are no objects are listed for reload from specific schema, then skip the reload feature or do not include the particular schema.QUEUE_TABLES
from reload.
1.7.2 PREMIGRATION ADVISOR COMPILATION FAILURES DURING DRY RUN - PRCZ-2103 CAN'T LOCATE JSON/PP.PM
Issue: The OPRCZ-2103 CAN'T LOCATE JSON/PP.PM
error occurs during the ZDM_PRE_MIGRATION_ADVISOR
phase.
Solution: When the source database is Oracle Database 11.2.0.4, for performing a logical migration, specify the following parameters in the response file:
RUNCPATREMOTELY=TRUE
COPYCPATREPORTTOZDMHOST=FALSE
1.7.3 ORA-23605: INVALID VALUE "" FOR GOLDENGATE PARAMETER PARALLELISM.
Issue: The Oracle GoldenGate Extract startup fails when the source database is Oracle Standard Edition 2, due to the following error:
ORA-23605: INVALID VALUE "" FOR GOLDENGATE PARAMETER PARALLELISM
.
Solution: If you do not apply the patch on the source database, then specify GOLDENGATESETTINGS_EXTRACT_PARALLELISM=1
parameter in the ZDM response file. ZDM will set TRANLOGOPTIONS INTEGRATEDPARAMS (parallelism 1)
for Oracle GoldenGate Extract.
1.7.4 PRCZ-4002 : failed to execute command "/bin/cp" using the privileged execution plugin "zdmauth" on nodes "dbserver"
Issue: The ZDMCLI RESUME JOB
command fails during migration and the ZDM job pauses at the ZDM_CONFIGURE_DG_SRC
phase. The error occurs when you update the /etc/hosts
file of the source database server with a different IP address or alias for the source database server.
Solution: Ensure that the IP address of the source database server is correctly updated in the /etc/hosts
file of the source database server and the ZDM server.
1.7.5 Migrating from Amazon Web Services RDS Oracle Database to Oracle Autonomous Database on Shared Exadata Infrastructure fails in validate phase with error PARAMETER ]] RHPHELPERUTIL-CONSTR-OHOME VALUE IS NOT VALID
Issue: Migrating from Amazon Web Services RDS Oracle Database to Oracle Autonomous Database on Shared Exadata Infrastructure fails in the ZDM_VALIDATE_SRC
phase with error PARAMETER]] RHPHELPERUTIL-CONSTR-OHOME VALUE IS NOT VALID
.
Solution: This issue occurs when -srcauth
details are provided in the command.
Remove the -srcauth
parameter when the source environment is Amazon Web Services RDS Oracle Database.
1.7.6 TLS Service is required for Fractional OCPU Services in Oracle Autonomous Database
Issue: The TLS service is required for fractional OCPU services in Oracle Autonomous Database service alias which is to be specified in the response file parameter. Specifying non-TLS alias is not supported.
Solution: If the target database is Oracle Autonomous Database on Dedicated Exadata Infrastructure or Oracle Autonomous Database on Exadata Cloud@Customer using fractional OCPU services, then you can specify TP_TLS
or LOW_TLS
aliases for the TARGETDATABASE_CONNECTIONDETAILS_SERVICENAME parameter.
For more information about specifying the requirement for the service alias for the target database, see Setting Logical Migration Parameters.
1.7.7 Migrating from AIX to EXACC using NFS with Non-readable Dump Fails to CHOWN
Issue: Migrating from AIX to EXACC using NFS with non-readable dump fails to CHOWN in source AIX host.
Solution: Use an alternate option for migrating using NFS which is documented in Migrating to Co-Managed Database Server with NFS Data Transfer Medium.
However the following scenario is not supported for IBM AIX: If the IDs do not match, Zero Downtime Migration automatically discovers the primary group of the target database user and changes the group of the dump to the primary group of the target database user.
1.7.8 Logical migration with DBUSER plugin must also set RUNCPATREMOTELY
Solution: To perform a logical migration using database user authentication plug-in as dbuser
, you must set value of the RUNCPATREMOTELY
parameter to TRUE
.
See RUNCPATREMOTELY for information about this parameter.
1.7.9 Warnings shown when running zdminstall
Issue: If home and base directories are not precreated, a warning similar to the following is shown when running the zdminstall.sh script.
/bin/df: '/ [...] /zdm21.3.1/home/..': No such file or directory/ [...] /zdm3rdparty/zdminstall.sh: line 2092: [: -lt: unary operator expected
Solution: This warning message can be ignored because the Zero Downtime Migration installer creates the home and base directories if they are not present. The warning does not affect the outcome of the installation or cause any issues for migration.
1.7.10 Warnings shown when running zdmservice operations
Issue: A warning similar to the following is shown when running zdmservice operations start
, stop
, status
, or deinstall
.
Use of uninitialized value in concatenation (.) or string at / [...] /zdm21.3.1/home/lib/jwcctl_lib.pm line 571.CRS_ERROR: Invalid data ALWAYS_ON= in _USR_ORA_ENV
Note that the line number in the output may vary.
Solution: This warning message can be ignored. It does not affect the use of the zdmservice
operations or cause any issues for migration.
1.7.11 Logical Migration Using DBLINK Fails with PRGZ-1177
Issue: "PRGZ-1177 : Database link "dblink_name" is invalid and unusable" error causes failure in a logical migration using a database link in a PDB or multitenant database in version 12.1.0.x.
Solution: See 12c PDB or Multitenant Only: ORA-02085: Database Link "LINK_NAME_HERE" Connects To "TARGET_DB" (Doc ID 2344831.1)
1.7.12 PRGZ-1161 : Predefined database service "TP" does not exist
Issue: PRGZ-1161 : Predefined database service "TP" does not exist for Autonomous Database ocid is a known issue for fractional OCPU configuration
If you choose to configure ‘Fractional ADB’ (Fraction of OCPU per DB instead of integer OCPU) – this flavor does not provide standard service alias HIGH and
Solution: Set the RSP parameter TARGETDATABASE_CONNECTIONDETAILS_SERVICENAME
to LOW_TLS or TP_TLS
The available services are - ‘low’ or ‘low_tls’ for Autonomous Data Warehouse with fractional OCPU, and ‘tp’ or ‘tp_tls’ for Autonomous Transaction Processing with fractional OCPU.
1.7.13 PRGG-1043 : No heartbeat table entries were found for Oracle GoldenGate Replicat process
Issue: An online logical migration job can report error PRGG-1043: No heartbeat table entries were found for Oracle GoldenGate Replicat process process_name due to one of the following causes:
-
Initialization parameter
job_queue_processes
was set to zero in the source or target database.Solution: Run the following statements on the database:
show parameter job_queue_processes;alter system set job_queue_processes=100 scope=both;exec dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED','FALSE');
-
Scheduled job
GG_UPDATE_HEARTBEATS
is not active in the source database. -
The server hosting Oracle GoldenGate deployments has a different time zone than the source database.
Solution: First, do one of the following solutions:
-
Modify the time zone for the server hosting Oracle GoldenGate deployments, OR
-
Use the web UI for the Oracle GoldenGate deployment to add Extract parameter
TRANLOGOPTIONS SOURCE_OS_TIMEZONE
and restart Extract.For example, if the source database time zone is UTC-5, then set parameter
TRANLOGOPTIONS SOURCE_OS_TIMEZONE -5
. For more information, see TRANLOGOPTIONS in Reference for Oracle GoldenGate.
Then, ensure that the
DST_PRIMARY_TT_VERSION
property in the source database is up to date. -
1.7.14 Restore Fails When Source Uses WALLET_ROOT
Issue: Zero Downtime Migration does not currently handle the migration of the TDE wallet from the source database to the target when the source database is using the wallet_root
initialization parameter. Without the wallets available on the target database, the restore phase fails with an error similar to the following:
RMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============RMAN-00571: ===========================================================RMAN-03002: failure of restore command at 06/15/2021 07:35:11ORA-19870: error while restoring backup piece/rman_PRD1/ZDM/IQPCZDM/c-3999816841-20210614-00ORA-19913: unable to decrypt backup
Solution: Manually copy the wallet to the target and resume the job.
1.7.15 PRCZ-4026 Thrown During Migration to Oracle Database 19.10 Target
Issue: When attempting to migrate to an Oracle Database 19.10 home at target, the migration job fails at phase ZDM_FINALIZE_TGT
with error PRCZ-4026, because of Oracle Clusterware (OCW) Bug 31070231.
PRCZ-4026 : Resource ora.db_unique_name.db is already running on nodes node.
Solution: Apply the Backport Label Request (BLR) for Bug#32646135 to the target 19.10 dbhome to avoid the reported issue. Once the BLR is applied, you can resume the failed migration job to completion.
Precaution: For physical migrations, you can avoid this issue by ensuring that your target database home is not on Oracle Database 19.10.
1.7.16 Environments With Oracle 11.2.0.4 Must Apply Perl Patch
Issue: Before using Zero Downtime Migration, you must apply a PERL patch if your source database environment meets either of the following conditions.
- Clusterware environment with Oracle Grid Infrastructure 11.2.0.4
- Single instance environment with Oracle Database 11.2.0.4
Solution: Download and apply Perl patch version 5.28.2 or later. Ensure that both the source and target Oracle Database 11g home include the patch for BUG 30508206 - UPDATE PERL IN 11.2.0.4 DATABASE ORACLE HOME TO V5.28.2.
1.7.17 ORA-39006 Thrown During Logical Migration to Oracle Autonomous Database on Dedicated Exadata Infrastructure Over Database Link
Issue: When attempting to migrate a database to an Oracle Autonomous Database on Dedicated Exadata Infrastructure target over a database link, the migration job fails with error ORA-39006.
ORA-39006: internal error
Solution: This is a Data Pump issue that is being tracked with Bug 31830685. Do not perform logical migrations over a database link to Oracle Autonomous Database on Dedicated Exadata Infrastructure targets until the bug is fixed and the fix is applied to the Autonomous target database.
1.7.18 Zero Downtime Migration Service Fails To Start After Upgrade
Issue: The following scenario occurs:
-
Perform migration jobs with Zero Downtime Migration 19.7
-
Response files used in those jobs are removed
-
Upgrade to Zero Downtime Migration 21.1
-
Attempt to run a migration
The following errors are seen.
CRS_ERROR:TCC-0004: The container was not able to start.
CRS_ERROR:One or more listeners failed to start. Full details will be found in the appropriate container log fileContext [/rhp] startup failed due to previous errors sync_start failed with exit code 1.
A similar error is found in the log files located in zdm_installation_location/base/crsdata/hostname/rhp/logs/
.
Caused by: oracle.gridhome.container.GHException: Internal error:PRGO-3003 : Zero downtime migration (ZDM) template file /home/jdoe/zdm_mydb.rsp does not exist.
Solution: To recover, manually recreate the response files listed in the log, and place them in the location specified in the log.
1.8 Troubleshooting
If you run into issues, check here in case a solution is published. For each issue, a workaround is provided.
1.8.1 Installation Issues
1.8.1.1 INS-42505 Warning Shown During Installation
Issue: The following warning is shown during installation.
/stage/user/ZDM_KIT_relnumber>./zdminstall.sh setuporaclehome=/stage/user/grid oraclebase=/stage/user/baseziploc=/stage/user/ZDM_KIT_relnumber/rhp_home.zip -zdm---------------------------------------Unzipping shiphome to gridhome---------------------------------------Unzipping shiphome...Shiphome unzipped successfully..---------------------------------------##### Starting GridHome Software Only Installation #####---------------------------------------Launching Oracle Grid Infrastructure Setup Wizard...[WARNING] [INS-42505] The installer has detected that the Oracle GridInfrastructure home software at (/stage/user/grid) is not complete. CAUSE: Following files are missing:...
Solution: This warning message can be ignored. It does not affect the installation or cause any issues for migration.
1.8.2 Connectivity Issues
1.8.2.1 General Connectivity Issues
Issue: If connectivity issues occur between the Zero Downtime Migration service host and the source or target environments, or between source and target environments, check the following areas.
Solution: Verify that the SSH configuration file (/root/.ssh/config
) has the appropriate entries:
Host * ServerAliveInterval 10 ServerAliveCountMax 2Host ocidb1 HostName 192.0.2.1 IdentityFile ~/.ssh/ocidb1.ppk User opc ProxyCommand /usr/bin/nc -X connect -x www-proxy.example.com:80 %h %p
Note that the proxy setup might not be required when you are not using a proxy server for connectivity. For example, when the source database server is on Oracle Cloud Infrastructure Classic, you can remove or comment the line starting with ProxyCommand
.
If the source is an Oracle RAC database, then make sure you copy the ~/.ssh/config
file to all of the source Oracle RAC servers. The SSH configuration file refers to the first Oracle RAC server host name, public IP address, and private key attributes.
1.8.2.2 Communications Link Failure
Issue: If the MySQL server crashes you will see errors such as this one for the ZDM operations:
$ ./zdmcli query job -jobid 6Exception [EclipseLink-4002] (Eclipse Persistence Services -2.7.7.qualifier): org.eclipse.persistence.exceptions.DatabaseExceptionInternal Exception: com.mysql.cj.jdbc.exceptions.CommunicationsException:Communications link failureThe last packet sent successfully to the server was 0 milliseconds ago. Thedriver has not received any packets from the server.Error Code: 0Query: ReadAllQuery(referenceClass=JobSchedulerImpl sql="SELECTJOB_IDENTIFIER, M_ACELIST, ARGUMENTS, ATTRIBUTES, CLIENT_NAME,COMMAND_PROVIDED, COMPARTMENT, CONTAINER_TYPE, CREATEDATE, CREATOR,CURRENT_STATUS, DB_OCID, DBNAME, DEPLOYMENT_OCID, DISABLE_JOB_EXECUTION,ELAPSED_TIME, END_TIME, EXECUTE_PHASES, EXECUTION_TIME, IS_EVAL, IS_PAUSED,JOB_TYPE, METHOD_NAME, METRICS_LOCATION, OPERATION, PARAMETERS,PARENT_JOB_ID, PAUSE_AFTER_PHASE, RESULT, PHASE, JOB_SCHEDULER_PHASES,REGION, REST_USER_NAME, RESULT_LOCATION, SCHEDULED_TIME, SITE, SOURCEDB,SOURCENODE, SOURCESID, SPARE1, SPARE2, SPARE3, SPARE_A, SPARE_B, SPARE_C,START_TIME, STOP_AFTER_PHASE, TARGETNODE, JOB_THREAD_ID, UPD_DATE, USER_NAME,ENTITY_VERSION, CUSTOMER FROM JOBSCHEDULER WHERE (PARENT_JOB_ID = ?)")
Solution: If such Communications errors are seen, restart the Zero Downtime Migration service so that the MySQL server is restarted, after which the pending jobs will resume automatically.
Stop the Zero Downtime Migration service:
zdmuser> $ZDM_HOME/bin/zdmservice stop
Start the Zero Downtime Migration service:
zdmuser> $ZDM_HOME/bin/zdmservice start
1.8.2.3 Evaluation Fails in Phase ZDM_GET_TGT_INFO
Issue: During the evaluation (-eval
) phase of the migration process, the evaluation fails in the ZDM_GET_TGT_INFO
phase with the following error for the Oracle RAC instance migration.
Executing phase ZDM_GET_TGT_INFORetrieving information from target node "trac11" ...PRGZ-3130 : failed to establish connection to target listener from nodes [srac11, srac12]PRCC-1021 : One or more of the submitted commands did not execute successfully.PRCC-1025 : Command submitted on node srac11 timed out after 15 seconds.PRCC-1025 : Command submitted on node srac12 timed out after 15 seconds.
Solution:
- Get the SCAN name of source database and add it to the
/etc/hosts
file on both target database servers, with the public IP address of the source database server and the source database SCAN name. For example:192.0.2.3 source-scan
- Get the SCAN name of the target database and add it to the
/etc/hosts
file on both source database servers, with the public IP address of the target database server and target database SCAN name. For example:192.0.2.1 target-scan
Note:
This issue, where the SCAN IP address is not added to /etc/hosts
file, might occur because in some cases the SCAN IP address is assigned as a private IP address, so it might not be resolvable.
1.8.2.4 Object Storage Is Not Accessible
Issue: When Object Storage is accessed from the source or target database server, it may fail with the following error.
About to connect() to swiftobjectstorage.xx-region-1.oraclecloud.com port 443 (#0)Trying 192.0.2.1... No route to hostTrying 192.0.2.2... No route to hostTrying 192.0.2.3... No route to hostcouldn't connect to hostClosing connection #0curl: (7) couldn't connect to host
Solution: On the Zero Downtime Migration service host, in the response file template ($ZDM_HOME/rhp/zdm/template/zdm_template.rsp
), set the Object Storage Service proxy host and port parameters listed below, if a proxy is required to connect to Object Storage from the source database server. For example:
SRC_OSS_PROXY_HOST=www-proxy-source.example.comSRC_OSS_PROXY_PORT=80
In the response file template ($ZDM_HOME/rhp/zdm/template/zdm_template.rsp
), set the Object Storage Service proxy host and port parameters listed below, if a proxy is required to connect to Object Storage from the target database server. For example:
TGT_OSS_PROXY_HOST=www-proxy-target.example.comTGT_OSS_PROXY_PORT=80
1.8.2.5 SSH Error "EdDSA provider not supported"
Issue: The following error messages appear in $ZDM_BASE/crsdata/zdm service hostname/rhp/zdmserver.log.0
.
[sshd-SshClient[3051eb49]-nio2-thread-1] [ 2020-04-04 00:26:24.142 GMT ] [JSChChannel$LogOutputStream.flush:1520] 2020-04-04: WARNING: org.apache.sshd.client.session.C: globalRequest(ClientConnectionService[ClientSessionImpl[opc@samidb-db/140.238.254.80:22]])[hostkeys-00@openssh.com, want-reply=false] failed (SshException) to process: EdDSA provider not supported[sshd-SshClient[3051eb49]-nio2-thread-1] [ 2020-04-04 00:26:24.142 GMT ] [JSChChannel$LogOutputStream.flush:1520] 2020-04-04: FINE : org.apache.sshd.client.session.C: globalRequest(ClientConnectionService[ClientSessionImpl[opc@samidb-db/140.238.254.80:22]])[hostkeys-00@openssh.com, want-reply=false] failure detailsorg.apache.sshd.common.SshException: EdDSA provider not supported at org.apache.sshd.common.util.buffer.Buffer.getRawPublicKey(Buffer.java:446) at org.apache.sshd.common.util.buffer.Buffer.getPublicKey(Buffer.java:420) at org.apache.sshd.common.global.AbstractOpenSshHostKeysHandler.process(AbstractOpenSshHostKeysHandler.java:71) at org.apache.sshd.common.global.AbstractOpenSshHostKeysHandler.process(AbstractOpenSshHostKeysHandler.java:38) at org.apache.sshd.common.session.helpers.AbstractConnectionService.globalRequest(AbstractConnectionService.java:723) at org.apache.sshd.common.session.helpers.AbstractConnectionService.process(AbstractConnectionService.java:363) at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:400) at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:333) at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1097) at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:294) at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63) at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:357) at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:335) at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:332) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38) at java.security.AccessController.doPrivileged(Native Method) at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker$2.run(Invoker.java:218) at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.security.NoSuchAlgorithmException: EdDSA provider not supported at org.apache.sshd.common.util.security.SecurityUtils.generateEDDSAPublicKey(SecurityUtils.java:596) at org.apache.sshd.common.util.buffer.keys.ED25519BufferPublicKeyParser.getRawPublicKey(ED25519BufferPublicKeyParser.java:45) at org.apache.sshd.common.util.buffer.keys.BufferPublicKeyParser$2.getRawPublicKey(BufferPublicKeyParser.java:98) at org.apache.sshd.common.util.buffer.Buffer.getRawPublicKey(Buffer.java:444) ... 22 more[sshd-SshClient[3051eb49]-nio2-thread-1] [ 2020-04-04 00:26:24.142 GMT ] [JSChChannel$LogOutputStream.flush:1520] 2020-04-04: FINE : org.apache.sshd.client.session.C: sendGlobalResponse(ClientConnectionService[ClientSessionImpl[opc@samidb-db/140.238.254.80:22]])[hostkeys-00@openssh.com] result=ReplyFailure, want-reply=false[sshd-SshClient[3051eb49]-nio2-thread-2] [ 2020-04-04 00:26:24.182 GMT ] [JSChChannel$LogOutputStream.flush:1520] 2020-04-04: FINE : org.apache.sshd.common.io.nio2.N: handleReadCycleCompletion(Nio2Session[local=/192.168.0.2:41198, remote=samidb-db/140.238.254.80:22]) read 52 bytes
Solution: Zero Downtime Migration uses the RSA format.
1.8.3 Transparent Data Encryption Related Issues
1.8.3.1 Transparent Data Encryption General Information
Depending on your source database release, Transparent Data Encryption (TDE) wallet configuration may be required.
- Oracle Database 12c Release 2 and later
For Oracle Database 12c Release 2 and later releases, TDE wallet configuration is mandatory and must be enabled on the source database before migration begins.
If TDE is not enabled, the database migration will fail.
Upon restore, the database tablespaces are encrypted using the wallet.
- Oracle Database 12c Release 1 and earlier
On Oracle Database 12c Release 1 and Oracle Database 11g Release 2 (11.2.0.4), TDE configuration is not required.
For information about the behavior of TDE in an Oracle Cloud environment, see My Oracle Support document Oracle Database Tablespace Encryption Behavior in Oracle Cloud (Doc ID 2359020.1).
1.8.3.2 Job Fails in Phase ZDM_SETUP_TDE_TGT
Issue: The phase ZDM_SETUP_TDE_TGT
fails with one of the following errors.
Executing phase ZDM_SETUP_TDE_TGTSetting up Oracle Transparent Data Encryption (TDE) keystore on the target node oci1121 ...oci1121: <ERR_FILE><Facility>PRGZ</Facility><ID>ZDM_KEYSTORE_NOT_SETUP_ERR</ID><ARGS><ARG>oci112_phx1z3</ARG></ARGS></ERR_FILE>PRGO-3007 : failed to migrate database "db11204" with zero downtimePRCZ-4002 : failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" using the privileged execution plugin "zdmauth" on nodes "oci1121"PRCZ-2103 : Failed to execute command "/u01/app/18.0.0.0/grid/perl/bin/perl" on node "oci1121" as user "root". Detailed error:<ERR_FILE><Facility>PRGZ</Facility><ID>ZDM_KEYSTORE_NOT_SETUP_ERR</ID><ARGS><ARG>oci112_phx1z3</ARG></ARGS></ERR_FILE>
Error at target server in /tmp/zdm749527725/zdm/log/mZDM_oss_standby_setup_tde_tgt_71939.log2019-06-13 10:00:20: Keystore location /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME does not exists for database 'oci112_region'2019-06-13 10:00:20: Reporting error:<ERR_FILE><Facility>PRGZ</Facility><ID>ZDM_KEYSTORE_NOT_SETUP_ERR</ID><ARGS><ARG>oci112_region</ARG></ARGS></ERR_FILE>
Solution:
- Oracle Database 12c Release 1 and later
On the target database, make sure that
$ORACLE_HOME/network/admin/sqlnet.ora
points to the correct location of the TDE wallet. For exmaple:ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)
- Oracle Database 11g Release 2 (11.2.0.4) only
On the target database, make sure that
$ORACLE_HOME/network/admin/sqlnet.ora
points to the correct location of the TDE wallet, and replace the$ORACLE_UNQNAME
variable with the value obtained from theSHOW PARAMETER DB_UNIQUE_NAME
SQL command.For example, run
SQL> show parameter db_unique_namedb_unique_name string oci112_region
and replace
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))
with
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/oci112_region)))
1.8.4 Full Backup Phase (ZDM_BACKUP_FULL_SRC) Issues
1.8.4.1 Backup Fails with ORA-19836
Issue: Source database full backup fails with one of the following errors.
</ERRLINE><ERRLINE>ORA-19836: cannot use passphrase encryption for this backup</ERRLINE><ERRLINE>RMAN-03009: failure of backup command on C8 channel at 04/29/2019 20:42:16
</ERRLINE><ERRLINE>ORA-19836: cannot use passphrase encryption for this backup</ERRLINE><ERRLINE>RMAN-03009: continuing other job steps, job failed will not be re-run
Solution 1: This issue can occur if you specify the -sourcedb
value in the wrong case. For example, if the value obtained from SQL command SHOW PARAMETER DB_UNIQUE_NAME
is zdmsdb
, then you need to specify it as zdmsdb
in lower case, and not as ZDMSDB
in upper case, as shown in the following example.
zdmuser> $ZDM_HOME/bin/zdmcli migrate database -sourcedb zdmsdb -sourcenode ocicdb1 -srcroot-targetnode ocidb1 -targethome /u01/app/oracle/product/12.1.0.2/dbhome_1-backupuser backup_user@example.com -rsp /u01/app/zdmhome/rhp/zdm/template/zdm_template_zdmsdb.rsp-tgtauth zdmauth -tgtarg1 user:opc-tgtarg2 identity_file:/home/zdmuser/.ssh/zdm_service_host.ppk-tgtarg3 sudo_location:/usr/bin/sudo
Solution 2: For Oracle Database 12c Release 1 and later, ensure that $ORACLE_HOME/network/admin/sqlnet.ora
points to the correct location of the TDE wallet, as shown here.
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))
For Oracle Database 11g Release 2 (11.2.0.4) only, ensure that $ORACLE_HOME/network/admin/sqlnet.ora
points to the correct location of the TDE wallet as shown below, and replace the variable $ORACLE_UNQNAME
with the value obtained with the SQL statement SHOW PARAMETER DB_UNIQUE_NAME
.
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))
For example:
SQL> show parameter db_unique_namedb_unique_name string oci112_region
ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/oci112_region)))
Solution 3: Run the following query and make sure that the wallet status is OPEN
.
SQL> select * from v$encryption_walletWRL_TYPE-------------WRL_PARAMETER-------------STATUS-------------file/opt/oracle/dcs/commonstore/wallets/tde/abc_testOPEN
1.8.4.2 Backup Fails with ORA-19914 and ORA-28365
Issue: Source database full backup fails with the following errors.
channel ORA_SBT_TAPE_3: backup set complete, elapsed time: 00:00:15channel ORA_SBT_TAPE_3: starting compressed full datafile backup setchannel ORA_SBT_TAPE_3: specifying datafile(s) in backup setinput datafile file number=00005 name=+DATA/ODA122/7312FA75F2B202E5E053050011AC5977/DATAFILE/system.382.1003858429channel ORA_SBT_TAPE_3: starting piece 1 at 25-MAR-19RMAN-03009: failure of backup command on ORA_SBT_TAPE_3 channel at 03/25/2019 19:09:30ORA-19914: unable to encrypt backupORA-28365: wallet is not opencontinuing other job steps, job failed will not be re-runchannel ORA_SBT_TAPE_3: starting compressed full datafile backup setchannel ORA_SBT_TAPE_3: specifying datafile(s) in backup set
Solution: Ensure that the wallet is opened in the database, and in case of CDB, ensure that the wallet is opened in the CDB, all PDBs, and PDB$SEED. See Setting Up the Transparent Data Encryption Wallet in the Zero Downtime Migration documentation for information about setting up TDE.
1.8.4.3 Either the Bucket Named Object Storage Bucket Name Does Not Exist in the Namespace Namespace or You Are Not Authorized to Access It
See Oracle Support Knowledge Base article "Either the Bucket Named '<Object Storage Bucket Name>' Does not Exist in the Namespace '<Namespace>' or You are not Authorized to Access it (Doc ID 2605518.1)" for the desciption and workarounds for this issue.
1.8.5 Restore Phase (ZDT_CLONE_TGT) Issues
1.8.5.1 Restore Database Fails With Assert [KCBTSE_ENCDEC_TBSBLK_1]
Issue: Due to RDBMS Bugs 31048741, 32697431, and 32117834 you may see assert [kcbtse_encdec_tbsblk_1] in the alert log during restore phase of a physical migration.
Solution: Apply patches for RDBMS Bugs 31048741 and 32697431 to any Oracle Database 19c migration target previous to the 19.13 update.
1.8.5.2 Restore Database Fails With AUTOBACKUP does not contain an SPFILE
Issue: During the execution of phase ZDT_CLONE_TGT
, restore database fails with the following error.
channel C1: looking for AUTOBACKUP on day: 20200427channel C1: AUTOBACKUP found: c-1482198272-20200427-12channel C1: restoring spfile from AUTOBACKUP c-1482198272-20200427-12channel C1: the AUTOBACKUP does not contain an SPFILE
The source database is running using init.ora
file, but during the restore target phase, the database is trying to restore the server parameter file (SPFILE) from autobackup, therefore it fails.
Solution: Start the source database using an SPFILE and resubmit the migration job.
1.8.5.3 Restore Database Fails With ORA-01565
Issue: During the execution of phase ZDT_CLONE_TGT
, restore database fails with the following error.
</ERRLINE><ERRLINE>With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP</ERRLINE><ERRLINE>and Real Application Testing options</ERRLINE><ERRLINE></ERRLINE><ERRLINE>CREATE PFILE='/tmp/zdm833428275/zdm/PFILE/zdm_tgt_mclone_nrt139.pfile' FROM SPFILE</ERRLINE><ERRLINE>*</ERRLINE><ERRLINE>ERROR at line 1:</ERRLINE><ERRLINE>ORA-01565: error in identifying file '?/dbs/spfile@.ora'</ERRLINE><ERRLINE>ORA-27037: unable to obtain file status</ERRLINE><ERRLINE>Linux-x86_64 Error: 2: No such file or directory</ERRLINE><ERRLINE>Additional information: 3</ERRLINE><ERRLINE></ERRLINE><ERRLINE></ERRLINE><ERRLINE>Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production</ERRLINE><ERRLINE>With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP
Solution: Start the target database using an SPFILE and resume the migration job.
1.8.6 Post Migration Automatic Backup Issues
1.8.6.1 Troubleshooting Post Migration Automatic Backup Failures
Issue: Post migration, on the target database, Automatic Backup might fail.
You can verify the failure using the console in Bare Metal, VM and Exadata > DB Systems > DB System Details > Database Details > Backups.
Solution: Get the RMAN configuration settings from one of the following places.
- Zero Downtime Migration documentation in Target Database Prerequisites, if captured
- The log files at
/opt/oracle/dcs/log/hostname/rman/bkup/db_unique_name/
/tmp/zdmXXX/zdm/zdm_TDBNAME_rman.dat
For example, using the second option, you can get the RMAN configuration settings from /opt/oracle/dcs/log/ocidb1/rman/bkup/ocidb1_abc127/rman_configure*.log
, then reset any changed RMAN configuration settings for the target database to ensure that automatic backup works without any issues.
If this workaround does not help, then debug further by getting the RMAN job ID by running the DBCLI command, list-jobs
, and describe the job details for more error details by running the DBCLI command describe-job -i JOB ID
from the database server as the root user.
For example, during the test, the following highlighted settings were modified to make Automatic Backup work.
rman target /Recovery Manager: Release 12.2.0.1.0 - Production on Mon Jul 8 11:00:18 2019Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved.connected to target database: ORCL (DBID=1540292788)RMAN> show all;using target database control file instead of recovery catalogRMAN configuration parameters for database with db_unique_name OCIDB1_ABC127 are:CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;CONFIGURE BACKUP OPTIMIZATION OFF;CONFIGURE DEFAULT DEVICE TYPE TO DISK; # defaultCONFIGURE CONTROLFILE AUTOBACKUP ON;CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # defaultCONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # defaultCONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4 BACKUP TYPE TO COMPRESSED BACKUPSET;CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # defaultCONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # defaultCONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # defaultCONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # defaultCONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # defaultCONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 2 G;CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' MAXPIECESIZE 2 G FORMAT '%d_%I_%U_%T_%t' PARMS 'SBT_LIBRARY=/opt/oracle/dcs/commonstore/pkgrepos/oss/odbcs/libopc.so ENV=(OPC_PFILE=/opt/oracle/dcs/commonstore/objectstore/opc_pfile/1245080042/opc_OCIDB1_ABC127.ora)';CONFIGURE MAXSETSIZE TO UNLIMITED; # defaultCONFIGURE ENCRYPTION FOR DATABASE ON;CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # defaultCONFIGURE COMPRESSION ALGORITHM 'MEDIUM' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE;CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # defaultCONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE';CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+RECO/ OCIDB1_ABC127/controlfile/snapcf_ocidb1_abc127.f';CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK clear;RMAN>
1.8.6.2 Post Migration Automatic Backup Fails With DCS-10045
Issue: Post migration, Automatic Backup fails with the following error for non-TDE enabled migrated Oracle Database releases 11.2.0.4 and 12.1.0.2.
DCS-10045: Validation error encountered: Backup password is mandatory to take OSS backup for non-tde enabled database...
You can verify this error by getting the RMAN job ID by running DBCLI command list-jobs
, and describe the job details to get the error details by running DBCLI command describe-job -i JOB ID
from the database server as the root user.
Solution:
- Find the TDE wallet location.
The Oracle Cloud Infrastructure provisioned database instance will have following entry in
sqlnet.ora
.ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME)))
- Remove the
cwallet.sso
file from the wallet location.For example,
/opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME
. - For Oracle Database 11g Release 2, do the folowing steps.
- Connect to database using SQL*Plus as sysdba and verify the current wallet location.
SQL> select * from v$encryption_wallet;WRL_TYPE WRL_PARAMETER STATUSfile /opt/oracle/dcs/commonstore/wallets/tde/ocise112_region OPEN
- Close the wallet in the database.
SQL> alter system set wallet close;
- Open the wallet using the wallet password.
SQL> alter system SET WALLET open IDENTIFIED BY "walletpassword"
- Set the master encryption key.
SQL> alter system set encryption key identified by "walletpassword"
- Recreate the autologin SSO file.
/home/oracle>orapki wallet create -wallet /opt/oracle/dcs/commonstore/wallets/tde/$ORACLE_UNQNAME -auto_loginOracle PKI Tool : Version 11.2.0.4.0 - ProductionCopyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.Enter wallet password: #
- Retry Automatic Backup.
- Connect to database using SQL*Plus as sysdba and verify the current wallet location.
- For Oracle Database 12c, do the folowing steps.
- Connect to database using SQL*Plus as sysdba and verify the current wallet location and status.
SQL> SELECT wrl_parameter, status, wallet_type FROM v$encryption_wallet;WRL_PARAMETER STATUS WALLET_TYPE/opt/oracle/dcs/commonstore/wallets/tde/ocise112_region OPEN_NO_MASTER_KEY OPEN
If the STATUS column contains a value of OPEN_NO_MASTER_KEY, you must create and activate the master encryption key.
- Close the wallet in the database.
SQL> alter system set wallet close;
- Open the wallet-using password.
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE open IDENTIFIED BY "walletpassword" CONTAINER=all;
- Set the master encryption key.
SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "walletpassword" with backup;
Log in to each PDB and run
SQL> ALTER SESSION SET CONTAINER = PDB_NAME;SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "walletpassword" with backup;
- Create the auto login keystore.
SQL> ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE 'path to wallet directory' IDENTIFIED BY "walletpassword";
- Retry Automatic Backup.
- Connect to database using SQL*Plus as sysdba and verify the current wallet location and status.
1.8.6.3 Post Migration Automatic Backup Fails With DCS-10096
Issue: Post migration, Automatic Backup fails with the following error.
DCS-10096:RMAN configuration 'Retention policy' must be configured as 'configure retentio n policy to recovery window of 30 days'
You can verify this error by getting the RMAN job ID by running DBCLI command list-jobs
, and describe the job details for more error details by running DBCLI command describe-job -i JOB ID
from the database server as the root user.
Solution: Log in into RMAN prompt and configure the retention policy.
[oracle@racoci1 ~]$ rman target /Recovery Manager: Release 12.2.0.1.0 - Production on Wed Jul 17 11:04:35 2019Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved.connected to target database: SIODA (DBID=2489657199)RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;old RMAN configuration parameters:CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;new RMAN configuration parameters:CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 30 DAYS;new RMAN configuration parameters are successfully stored
Retry Automatic Backup.
1.8.7 Miscellaneous Issues
1.8.7.1 Migration from Existing Data Guard Standby Fails
Issue: Using an existing standby, Zero Downtime Migration job fails when Data Guard broker configuration uses TNS aliases.
In a Data Guard broker configuration, every database needs to be reachable from every other database in the configuration. When Zero Downtime Migration creates a new standby at the target and adds it to the existing Data Guard broker configuration, Zero Downtime Migration adds the target with connect identifier specified in the form of the connect string. Zero Downtime Migration does not update the tnsnames.ora on the target with other databases is in the configuration. Because the tnsnames.ora entries are missing, other databases may not be reachable if the configuration was created with TNS aliases.
Solution: Ensure that all TNS aliases in the broker configuration corresponding to the primary and any existing standby databases are defined in the target tnsnames.ora file.
Alternatively, ensure that the broker configuration is made up of connect strings instead of TNS aliases. The connect identifier string can be identified using the command below:
show database db_name dgconnectidentifier;
If the connect identifier is a TNS alias, the identifier can be updated using the command below and specifying the connect string in the form of EZconnect string.
For cluster databases:
edit database db_name set property dgconnectidentifier='scan_name:scan_port/service_name';
For non cluster database:
edit database db_name set propertydgconnectidentifier='listener_host:listener_port/service_name';
The TNS aliases are not required once the connect identifiers are specified as connect strings that are reachable from every database instance in the broker configuration. This is because the broker needs to be able to manage the primary/standby relationship in case any standby switches roles and becomes the primary.
1.8.7.2 PDB in Failed State After Migration to ExaCS or ExaCC
Issue: ExaCS and ExaCC recently added functionality to display the PDBs of the CDB. When the target database is provisioned with the same PDB name as the source before the migration, then after the migration, the PDB names report status as failed.
This is because when the target is provisioned the PDBID of the PDB is different. During the migration, Zero Downtime Migration drops the target and recreates it. So if the PDB names were the same but now have different internal PDBIDs, the control plane reports the PDB as failed.
Solutions: To avoid this problem, when provisioning the target:
-
If the source is non-CDB, provision a non-CDB target through dbaascli
-
If the source is a CDB with PDBs, provision the target without any PDBs
If the PDB is reported in the failed state post migration, the resolution would be to follow Pluggable Database(PDB) Resource Shows Failed Status In Cloud Console while it is Available in VM (Doc ID 2855062.1).
1.8.7.3 Oracle GoldenGate Hub Certificate Known Issues
Issue: Oracle Zero Downtime Migration leverages Oracle GoldenGate for its logical online migration work flow; an Oracle GoldenGate hub is set up on OCI compute for this purpose.
The Oracle GoldenGate hub NginX Reverse Proxy uses a self-signed certificate which will cause the following error:
SunCertPathBuilderException: unable to find valid certification path to requested target when ZDM Server makes a REST API call.
Solution: See My Oracle Support document Zero Downtime Migration - GoldenGate Hub Certificate Known Issues (Doc ID 2768483.1)
1.8.7.4 Source Discovery Does Not Find 'cut' in Default Location
Issue: Discovery at the source database server fails to find cut
in the standard location.
The source database deployment's standard cut
location is /bin/cut
. If cut
is not in the location, Zero Downtime Migration cannot discover the source database information correctly, and the migration fails in its initial phases.
Solution: To resolve the issue, ensure that cut
is installed in the standard /bin/cut
path or create a symbolic link to the installed location, for example:
ln -sf <installed_location_of_the_cut> /bin/cut
1.8.7.5 Evaluation Fails in Phase ZDM_GET_SRC_INFO
Issue: During the evaluation (-eval
) phase of the migration process, the evaluation fails in the ZDM_GET_SRC_INFO
phase with the following error for the source single instance deployed without Grid infrastructure.
Executing phase ZDM_GET_SRC_INFOretrieving information about database "zdmsidb" ...PRCF-2056 : The copy operation failed on node: "zdmsidb".Details: {1}PRCZ-4002 : failed to execute command "/bin/cp" using the privilegedexecution plugin "zdmauth" on nodes "zdmsidb"scp: /etc/oratab: No such file or directory
Solution: Make an ORACLE_HOME
value entry in file /etc/oratab
with value db_name:$ORACLE_HOME:N
, as shown in this example.
zdmsidb:/u01/app/oracle/product/12.2.0.1/dbhome_1:N
1.8.7.6 Migration Evaluation Failure with Java Exception Invalid Key Format
Issue: The following conditions are seen:
-
Zero Downtime Migration
migration -eval
command fails with the following error.Result file path contents:"/u01/app/zdmbase/chkbase/scheduled/job-19-2019-12-02-03:46:19.log"zdm-server.ocitoolingsn.ocitooling.oraclevcn.com: Processing responsefile ...null
-
The file
$ZDM_BASE/<zdm service host>/rhp/rhpserver.log.0
contains the following entry.Verify below error message observed in file $ZDM_BASE/<zdm servicehost>/rhp/rhpserver.log.0rhpserver.log.7:[pool-58-thread-1] [ 2019-12-02 02:08:15.178 GMT ][JSChChannel.getKeyPair:1603] Exception :java.security.spec.InvalidKeySpecException:java.security.InvalidKeyException: invalid key format
-
The Zero Downtime Migration installed user (For example: zdmuser) private key (id_rsa) file has the following entries.
-----BEGIN OPENSSH PRIVATE KEY----------MIIEogIBAAKCAQEAuPcjftR6vC98fAbU4FhYVKPqc0CSgibtMSouo1DtQ06ROPN0XpIEL4r8nGp+c5GSDONyhf0hiltBzg0fyqyurSw3XfGJq2Q6EQ61aL95Rt9CZh6bJSUwc69T4rHjvRnK824k4UpfUIqafOXb2mRgGVUkldo4yy+pLoGq1GwbsIYbS4tkuaYPKZ3A3H9ZA7MtZ5M0sNqnk/4Qy0d8VONWozxOLFC2A8zbbe7GdQw9khVqDb/xEND OPENSSH PRIVATE KEY-----
Solution: Authentication key pairs (private and public key) are not generated using the ssh-keygen
utility, so you must generate authentication key pairs using steps in Generating a Private SSH Key Without a Passphrase.
After generating authentication key pairs, the private key file content looks like the following.
-----BEGIN RSA PRIVATE KEY-----MIIEogIBAAKCAQEAuPcjftR6vC98fAbU4FhYVKPqc0CSgibtMSouo1DtQ06ROPN0XpIEL4r8nGp+c5GSDONyhf0hiltBzg0fyqyurSw3XfGJq2Q6EQ61aL95Rt9CZh6bJSUwc69T4rHjvRnK824k4UpfUIqafOXb2mRgGVUkldo4yy+pLoGq1GwbsIYbS4tkuaYPKZ3A3H9ZA7MtZ5M0sNqnk/4Qy0d8VONWozxOLFC2A8zbbe7GdQw9khVqDb/x-----END RSA PRIVATE KEY-----
Set up connectivity with the newly generated authentication key pairs and resume the migration job.
1.8.7.7 Migration Evaluation Fails with Error PRCG-1022
Issue: The following conditions are seen:
$ZDM_HOME/bin/zdmcli migrate database -sourcedb zdmsdb -sourcenode ocicdb1 -srcauth zdmauth -srcarg1 user:opc -srcarg2 identity_file:/home/zdmuser/.ssh/zdm_service_host.ppk -srcarg3 sudo_location:/usr/bin/sudo -targetnode ocidb1 -backupuser backup_user@example.com -rsp /u01/app/zdmhome/rhp/zdm/template/zdm_template_zdmsdb.rsp -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/zdm_service_host.ppk -tgtarg3 sudo_location:/usr/bin/sudo -evalPRCG-1238 : failed to execute the Rapid Home Provisioning action for command 'migrate database'PRCG-1022 : failed to connect to the Rapid Home Provisioning daemon for cluster anandutestFailed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException[Root exception is java.rmi.ConnectException: Connection refused to host:anandutest; nested exception is: java.net.ConnectException: Connection refused (Connection refused)]
Solution: Start the Zero Downtime Migration service using the $ZDM_HOME/bin/zdmservice START
command, then run any ZDMCLI
commands.
1.8.7.8 ORA-01031 on Full Export from an Oracle 12.1 Source
Issue: When performing a full database export with Export Data Pump from an Oracle Database 12c (12.1) source database, the following errors occur:
05-AUG-21 10:36:12.483: ORA-31693: Table data object "SYS"."TABLE" failed to load/unload and is being skipped due to error: ORA-01031: insufficient privileges
Solution: See My Oracle Support document EXPDP - ORA-31693 ORA-01031 (Insufficient Privileges) On Some Tables When Exporting from 12cR1 (Doc ID 1676411.1)
1.8.7.9 Data Transfer Medium COPY Issues
Issue: Migrating data using logical migration with DATA_TRANSFER_MEDIUM=COPY
set in the Zero Downtime Migration response file fails.
Solution: When you specify DATA_TRANSFER_MEDIUM=COPY
you must also specify the following DUMPTRANSFERDETAILS_SOURCE_*
parameters.
DUMPTRANSFERDETAILS_TRANSFERTARGET_DUMPDIRPATH=<Target path to transferthe dumps to >DUMPTRANSFERDETAILS_TRANSFERTARGET_HOST=<Target Db server or Target sidetransfer node >DUMPTRANSFERDETAILS_TRANSFERTARGET_USER=<user having write access to specified path>DUMPTRANSFERDETAILS_TRANSFERTARGET_USERKEY=<user authentication keypath on zdm node>
1.8.7.10 Unable to Rerun MIGRATE DATABASE Command
Issue: Zero Downtime Migration blocks attempts to rerun the MIGRATE DATABASE
command for a specified database if that database is already part of an ongoing migration job.
Workaround: If you want to resubmit a database migration, you can stop the ongoing migration job in either EXECUTING
or PAUSED
state using the ZDMCLI ABORT JOB
command as follows.
-bash-4.2$ ./zdmcli abort job -jobid 70server.example.com: Audit ID: 189
1.8.7.11 Unable to Resume a Migration Job
Issue: Zero Downtime Migration writes the source and target log files to the /tmp/zdm-unique id
directory in the respective source and target database servers.
If you pause a migration job and and then resume the job after several (sometimes 15-20 days), the /tmp/zdm-unique id
directory might be deleted or purged as part of a clean up or server reboot that also cleans up /tmp.
Solution: After pausing a migration job, back up the /tmp/zdm-unique id
directory. Before resuming the migration job, check the /tmp
directory for /zdm-unique id
, and if it is missing, restore the directory and its contents with your backup.
1.8.7.12 Migration Job Fails at ZDM_GET_SRC_INFO
Issue: A migration job fails with the following error.
[opc@zdm-server rhp]$ cat /home/opc/zdm_base/chkbase/scheduled/job-34-2021-01-23-14:10:32.logzdm-server: 2021-01-23T14:10:32.155Z : Processing response file ...zdm-server: 2021-01-23T14:10:32.262Z : Starting zero downtime migrate operation ...PRCZ-4002 : failed to execute command "/bin/cp" using the privileged execution plugin "zdmauth" on nodes "PROD.compute-usconnectoneb95657.oraclecloud.internal"
Solution: You must set up SSH connectivity without a passphrase for the oracle user.
1.8.7.13 Migration Job Fails at ZDM_SWITCHOVER_SRC
Issue: A migration job fails at ZDM_SWITCHOVER_SRC
phase.
Solutions:
-
Ensure that there is connectivity from PRIMARY database nodes to STANDBY database nodes so the redo log are shipped as expected.
-
A job will fail at
ZDM_SWITCHOVER_SRC
if the recovery process (MRP0) is not running at the target. The recovery process reason for failure should be corrected if MRP0 is not running at Oracle Cloud Database Standby Instance, and then the process should be started manually at Oracle Cloud Database Standby Instance before the migration job can be resumed.
1.9 Additional Information for Migrating to Oracle Exadata Database Service
Read the following for general information, considerations, and links to more information about using Zero Downtime Migration to migrate your database to Oracle Exadata Database Service on Dedicated Infrastructure.
1.9.1 Considerations for Migrating to Oracle Exadata Database Service on Dedicated Infrastructure
For this release of Zero Downtime Migration be aware of the following considerations.
- If the source database is release 18c, then the target home should be at release 18.6 or later to avoid issues such as Bug 29445548 Opening Database In Cloud Environment Fails With ORA-600.
- If a backup was performed when one of the configured instances is down, you will encounter Bug 29863717 - DUPLICATING SOURCE DATABASE FAILED BECAUSE INSTANCE 1 WAS DOWN.
- The TDE keystore password must be set in the credential wallet. To set the password as part of the Zero Downtime Migration workflow, specify the
-tdekeystorewallet tde_wallet_path
or-tdekeystorepasswd
argument irrespective of whether the wallet usesAUTOLOGIN
orPASSWORD
. In either case the password is stored in the credential wallet. If the-tdekeystorepasswd
argument is not supplied, then Zero Downtime Migration skips the settingtde_ks_passwd
key in the credential wallet, and no error is thrown. - The target environment must be installed with latest DBaaS Tooling RPM with
db_unique_name
change support to be installed. - Provision a target database from the console without enabling auto-backups. In the Configure database backups section do not select the Enable automatic backups option.
1.9.2 Oracle Exadata Database Service on Dedicated Infrastructure Database Registration
Post migration, register the Oracle Exadata Database Service on Dedicated Infrastructure database, and make sure its meets all of the requirements.
Run the following commands on the Oracle Exadata Database Service on Dedicated Infrastructure database server as the root user.
/root>dbaascli registerdb prereqs --dbname db_name --db_unique_name db_unique_name/root>dbaascli registerdb begin --dbname db_name --db_unique_name db_unique_name
For example
/root>dbaascli registerdb prereqs --dbname ZDM122 --db_unique_name ZDM122_phx16nDBAAS CLI version 18.2.3.2.0Executing command registerdb prereqs --db_unique_name ZDM122_phx16nINFO: Logfile Location: /var/opt/oracle/log/ZDM122/registerdb/registerdb_2019-08-14_05:35:31.157978280334.logINFO: Prereqs completed successfully/root>/root>dbaascli registerdb begin --dbname ZDM122 --db_unique_name ZDM122_phx16nDBAAS CLI version 18.2.3.2.0Executing command registerdb begin --db_unique_name ZDM122_phx16nLogfile Location: /var/opt/oracle/log/ZDM122/registerdb/registerdb_2019-08-14_05:45:27.264851309165.logRunning prereqsDBAAS CLI version 18.2.3.2.0Executing command registerdb prereqs --db_unique_name ZDM122_phx16nINFO: Logfile Location: /var/opt/oracle/log/ZDM122/registerdb/registerdb_2019-08-14_05:45:29.000432309894.logINFO: Prereqs completed successfullyPrereqs completedRunning OCDE .. will take time ..OCDE Completed successfully.INFO: Database ZDM122 registered as Cloud database/root>
1.9.3 Oracle Exadata Database Service on Dedicated Infrastructure Automatic Backup Issues
Check the backup configuration before you enable automatic backup from the console. You can use the get config
command as shown in the first step below. You should see bkup_oss=no
before you enable automatic backup.
You might see the error message in the console, "A backup configuration exists for this database. You must remove the existing configuration to use Oracle Cloud Infrastructure's managed backup feature."
To fix this error, remove the existing configuration.
First, make sure the automatic backup is disabled from the UI, then follow these steps to remove the existing backup configuration.
- Generate a backup configuration file.
/var/opt/oracle/bkup_api/bkup_api get config --file=/tmp/db_name.bk --dbname=db_name
For example:
/var/opt/oracle/bkup_api/bkup_api get config --file=/tmp/zdmdb.bk --dbname=zdmdb
- Open the /tmp/db_name.bk file you created in the previous step.
For example: Open /tmp/zdmdb.bk
change bkup_oss=yes from bkup_oss=no
- Disable OSS backup by setting
bkup_oss=no
./var/opt/oracle/bkup_api/bkup_api set config --file=/tmp/db_name.bk --dbname=db_name
For example:
/var/opt/oracle/bkup_api/bkup_api set config --file=/tmp/zdmdb.bk --dbname=zdmdb
- Check reconfigure status.
/var/opt/oracle/bkup_api/bkup_api configure_status --dbname=db_name
For example:
/var/opt/oracle/bkup_api/bkup_api configure_status --dbname=zdmdb
Now enable automatic backup from console.
Verify the backups from the console. Click Create Backup to create a manual backup, and a backup should be created without any issues. and also Automatic Backup should be successful.
1.10 Additional Information for Migrating to Oracle Exadata Database Service on Cloud@Customer
Read the following for general information, considerations, and links to more information about using Zero Downtime Migration to migrate your database to Oracle Exadata Database Service on Cloud@Customer.
1.10.1 Considerations for Migrating to Oracle Exadata Database Service on Cloud@Customer
For this release of Zero Downtime Migration be aware of the following considerations.
- You must apply the regDB patch for Bug 29715950 - "modify regdb to handle db_unique_name not same as db_name" on all Oracle Exadata Database Service on Cloud@Customer nodes. This is required for the
ZDM_MANIFEST_TO_CLOUD
phase. Please note that the regDB tool is part of DBaaS Tooling. - If the source database is release 18c, then the target home should be at release 18.6 or later to avoid issues such as Bug 29445548 Opening Database In Cloud Environment Fails With ORA-600.
- PDB conversion related phases are listed in
-listphases
and can be ignored. Those are no-op phases. - If the backup medium is Zero Data Loss Recovery Appliance, then all configured instances should be up at the source when a
FULL
orINCREMENTAL
backup is performed. - If a backup was performed when one of the configured instances is down, you will encounter Bug 29863717 - DUPLICATING SOURCE DATABASE FAILED BECAUSE INSTANCE 1 WAS DOWN.
- The TDE keystore password must be set in the credential wallet. To set the password as part of the Zero Downtime Migration workflow, specify the
-tdekeystorewallet tde_wallet_path
or-tdekeystorepasswd
argument irrespective of whether the wallet usesAUTOLOGIN
orPASSWORD
. In either case the password is stored in the credential wallet. If the-tdekeystorepasswd
argument is not supplied, then Zero Downtime Migration skips the settingtde_ks_passwd
key in the credential wallet, and no error is thrown. - The target environment must be installed with latest DBaaS Tooling RPM with
db_unique_name
change support to be installed.
1.11 Documentation Accessibility
For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at .
Access to Oracle Support
Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit or visit if you are hearing impaired.