As you know there are two types of cloud services AWS provides (EC2 & RDS) while EC2 let you have an Operating System OS access to the DB machine including root access, RDS doesn't give you any kind of OS access. In RDS the master admin user which AWS provides to you has least admin privileges (neither has SYSDBA nor DBA role) as the database is supposed to be maintained by AWS. Though using this user to perform simple admin tasks like import a schema is a bit challenging on RDS, without an OS access you won't be able to use commands like impdp or imp the thing will force you to start exploring the alternative Oracle packages which can do this job from inside the DB, and yes Oracle has many built-in packages allow you to perform lots of tasks without the need to have an OS access.
Actually, Amazon already well documented importing a schema into RDS in this link but I thought to explain it in more details like a real-world task:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html
Task Summary:
I'll be exporting "EPS_OWNER" schema on an 11.2.0.3 database resides on AWS EC2 Linux instance and upload the export dump file to S3 bucket, then import the dump file into a 12.2.0.1 AWS RDS database to "EPS" schema.
Prerequisites:
- An AWS S3 bucket must be created and Both Source EC2 and Target RDS must have RW access to it through a role. [S3 bucket is kind of shared storage between AWS cloud systems where you can upload/download the files to it, it will be used during this demo to share the export dump file between EC2 source instance and RDS target instance].
Step1: Export the schema on Source [EC2 instance]:
I already have access to OS oracle user on the source EC2 instance so I used exportdata script to export EPS_OWNER schema, it generated the pre and post scripts to run before and after the import task on the target, but because I'll import to a schema with a different name so I adjusted those scripts by replacing the source schema name "EPS_OWNER" with the target schema name "EPS".
Step2: Upload the export file to S3 Bucket from Source [EC2 instance]:
In case the bucket is not yet configured on the source machine you can use the following AWSCLI command to configure it providing the bucket's "Access Key" and "Secret Access Key":
# aws configure
AWS Access Key ID [None]: AEFETFWNINTIHMLBWII5Q
AWS Secret Access Key [None]: EdfefrgzA1+kEtfs2kg43RtdSv/Il/wwxtD6vthty
Default region name [None]:
Default output format [None]:
Upload the export dump files to the S3 bucket:
# cd /backup
# aws s3 cp EXPORT_eps_owner_STG_04-03-19.dmp s3://eps-bucket
Step2: Download the export file from the S3 Bucket to the Target [RDS instance]:
Use the AWS built-in package "rdsadmin.rdsadmin_s3_tasks" to download the dump file from S3 bucket to DATA_PUMP_DIR:
SELECT rdsadmin.rdsadmin_s3_tasks.download_from_s3(
p_bucket_name => 'eps-bucket',
p_directory_name => 'DATA_PUMP_DIR')
AS TASK_ID FROM DUAL;
It will return a TASK ID:
TASK_ID
--------------------------
1554286165468-636
-- Create a tablespace: [Using Oracle Managed Files OMF]
CREATE SMALLFILE TABLESPACE "TBS_EPS" DATAFILE SIZE 100M AUTOEXTEND ON NEXT 100M LOGGING EXTENT MANAGEMENT LOCAL AUTOALLOCATE SEGMENT SPACE MANAGEMENT AUTO;
-- Create the user: [Here the user as per my business requirements will be different than the original user on the Source DB]
CREATE USER EPS IDENTIFIED BY "test123" DEFAULT TABLESPACE TBS_EPS TEMPORARY TABLESPACE TEMP QUOTA UNLIMITED ON TBS_EPS PROFILE DEFAULT;
GRANT CREATE SESSION TO EPS;
GRANT CREATE JOB TO EPS;
GRANT CREATE PROCEDURE TO EPS;
GRANT CREATE SEQUENCE TO EPS;
GRANT CREATE TABLE TO EPS;
DECLARE
hdnl NUMBER;
BEGIN
hdnl := DBMS_DATAPUMP.OPEN( operation => 'IMPORT', job_mode => 'SCHEMA', job_name=>null);
DBMS_DATAPUMP.ADD_FILE( handle => hdnl, filename => 'EXPORT_eps_owner_STG_04-03-19.dmp', directory => 'DATA_PUMP_DIR', filetype => dbms_datapump.ku$_file_type_dump_file, reusefile => 1);
--DBMS_DATAPUMP.ADD_FILE( handle => hdnl, filename => 'EXPORT_eps_owner_STG_04-03-19.log', directory => 'DATA_PUMP_DIR', filetype => dbms_datapump.ku$_file_type_log_file);
--DBMS_DATAPUMP.METADATA_FILTER(hdnl,'SCHEMA_EXPR','IN (''EPS_OWNER'')');
--DBMS_DATAPUMP.SET_PARAMETER(hdnl,'TABLE_EXISTS_ACTION','SKIP');
DBMS_DATAPUMP.METADATA_REMAP(hdnl,'REMAP_SCHEMA','EPS_OWNER','EPS');
DBMS_DATAPUMP.START_JOB(hdnl);
END;
/
The hashed parameters in gray color are there for reference:
--DBMS_DATAPUMP.ADD_FILE( handle => hdnl, filename => 'EXPORT_eps_owner_STG_04-03-19.log', directory => 'DATA_PUMP_DIR', filetype => dbms_datapump.ku$_file_type_log_file);
In case you want to write the import operation log into a log file.
--DBMS_DATAPUMP.METADATA_FILTER(hdnl,'SCHEMA_EXPR','IN (''EPS_OWNER'')');
--DBMS_DATAPUMP.SET_PARAMETER(hdnl,'TABLE_EXISTS_ACTION','SKIP');
In case you are used wrong parameters or bad combination e.g. using METADATA_FILTER instead of METDATA_REMAP when you are importing to a schema with a different name, you will get a bunch of errors similar to the below cute unclear ones:
Check the progress of the imported objects:
select object_type,count(*) from dba_objects where owner='EPS' group by object_type;
Run the After Import script that generated by exportdata script at Step 1 after replacing the original exported schema name EPS_OWNER with the target imported schema name EPS.
Check the invalid objects:
col object_name for a45
select object_name,object_type,status from dba_objects where owner='EPS' and status<>'VALID';
Check the exist files under DATA_PUMP_DIRdirectory:
select * from table(RDSADMIN.RDS_FILE_UTIL.LISTDIR('DATA_PUMP_DIR')) order by mtime;
References:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html
S3 Bucket creation:
https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
DBMS_DATAPUMP:
https://docs.oracle.com/database/121/ARPLS/d_datpmp.htm#ARPLS356
RDS Master Admin User:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.MasterAccounts.html
Actually, Amazon already well documented importing a schema into RDS in this link but I thought to explain it in more details like a real-world task:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html
Task Summary:
I'll be exporting "EPS_OWNER" schema on an 11.2.0.3 database resides on AWS EC2 Linux instance and upload the export dump file to S3 bucket, then import the dump file into a 12.2.0.1 AWS RDS database to "EPS" schema.
Prerequisites:
- An AWS S3 bucket must be created and Both Source EC2 and Target RDS must have RW access to it through a role. [S3 bucket is kind of shared storage between AWS cloud systems where you can upload/download the files to it, it will be used during this demo to share the export dump file between EC2 source instance and RDS target instance].
Step1: Export the schema on Source [EC2 instance]:
I already have access to OS oracle user on the source EC2 instance so I used exportdata script to export EPS_OWNER schema, it generated the pre and post scripts to run before and after the import task on the target, but because I'll import to a schema with a different name so I adjusted those scripts by replacing the source schema name "EPS_OWNER" with the target schema name "EPS".
Step2: Upload the export file to S3 Bucket from Source [EC2 instance]:
In case the bucket is not yet configured on the source machine you can use the following AWSCLI command to configure it providing the bucket's "Access Key" and "Secret Access Key":
# aws configure
AWS Access Key ID [None]: AEFETFWNINTIHMLBWII5Q
AWS Secret Access Key [None]: EdfefrgzA1+kEtfs2kg43RtdSv/Il/wwxtD6vthty
Default region name [None]:
Default output format [None]:
Upload the export dump files to the S3 bucket:
# cd /backup
# aws s3 cp EXPORT_eps_owner_STG_04-03-19.dmp s3://eps-bucket
Step2: Download the export file from the S3 Bucket to the Target [RDS instance]:
Remember, there is no OS access on RDS, so we will connect to the database using any tools such as SQL Developer using the RDS master user.
Use the AWS built-in package "rdsadmin.rdsadmin_s3_tasks" to download the dump file from S3 bucket to DATA_PUMP_DIR:
SELECT rdsadmin.rdsadmin_s3_tasks.download_from_s3(
p_bucket_name => 'eps-bucket',
p_directory_name => 'DATA_PUMP_DIR')
AS TASK_ID FROM DUAL;
It will return a TASK ID:
TASK_ID
--------------------------
1554286165468-636
Use this TASK_ID to monitor the download progress by running this statement:
SELECT text FROM table(rdsadmin.rds_file_util.read_text_file('BDUMP','dbtask-1554286165468-636.log'));
Once the download complete, query the downloaded files under DATA_PUMP_DIR using this query:
select * from table(RDSADMIN.RDS_FILE_UTIL.LISTDIR('DATA_PUMP_DIR')) order by mtime;
Once the download complete, query the downloaded files under DATA_PUMP_DIR using this query:
select * from table(RDSADMIN.RDS_FILE_UTIL.LISTDIR('DATA_PUMP_DIR')) order by mtime;
Now the AWS related tasks are done, let's jump to the import part which is purely Oracle's.
Step3: Create the tablespace and the target schema user on the Target [RDS instance]:
In case the target user does not yet exist on the target RDS database, you can go ahead and create it along with its tablespace.-- Create a tablespace: [Using Oracle Managed Files OMF]
CREATE SMALLFILE TABLESPACE "TBS_EPS" DATAFILE SIZE 100M AUTOEXTEND ON NEXT 100M LOGGING EXTENT MANAGEMENT LOCAL AUTOALLOCATE SEGMENT SPACE MANAGEMENT AUTO;
-- Create the user: [Here the user as per my business requirements will be different than the original user on the Source DB]
CREATE USER EPS IDENTIFIED BY "test123" DEFAULT TABLESPACE TBS_EPS TEMPORARY TABLESPACE TEMP QUOTA UNLIMITED ON TBS_EPS PROFILE DEFAULT;
GRANT CREATE SESSION TO EPS;
GRANT CREATE JOB TO EPS;
GRANT CREATE PROCEDURE TO EPS;
GRANT CREATE SEQUENCE TO EPS;
GRANT CREATE TABLE TO EPS;
Step4: Import the dump file on the Target [RDS instance]:
By the RDS master user run:DECLARE
hdnl NUMBER;
BEGIN
hdnl := DBMS_DATAPUMP.OPEN( operation => 'IMPORT', job_mode => 'SCHEMA', job_name=>null);
DBMS_DATAPUMP.ADD_FILE( handle => hdnl, filename => 'EXPORT_eps_owner_STG_04-03-19.dmp', directory => 'DATA_PUMP_DIR', filetype => dbms_datapump.ku$_file_type_dump_file, reusefile => 1);
--DBMS_DATAPUMP.ADD_FILE( handle => hdnl, filename => 'EXPORT_eps_owner_STG_04-03-19.log', directory => 'DATA_PUMP_DIR', filetype => dbms_datapump.ku$_file_type_log_file);
--DBMS_DATAPUMP.METADATA_FILTER(hdnl,'SCHEMA_EXPR','IN (''EPS_OWNER'')');
--DBMS_DATAPUMP.SET_PARAMETER(hdnl,'TABLE_EXISTS_ACTION','SKIP');
DBMS_DATAPUMP.METADATA_REMAP(hdnl,'REMAP_SCHEMA','EPS_OWNER','EPS');
DBMS_DATAPUMP.START_JOB(hdnl);
END;
/
The hashed parameters in gray color are there for reference:
--DBMS_DATAPUMP.ADD_FILE( handle => hdnl, filename => 'EXPORT_eps_owner_STG_04-03-19.log', directory => 'DATA_PUMP_DIR', filetype => dbms_datapump.ku$_file_type_log_file);
In case you want to write the import operation log into a log file.
--DBMS_DATAPUMP.METADATA_FILTER(hdnl,'SCHEMA_EXPR','IN (''EPS_OWNER'')');
In case the exported schema "EPS_OWNER" will be imported on an already existing schema with the same name, which is not my case here, in case you will use this parameter you should NOT use DBMS_DATAPUMP.METADATA_REMAP with it.
--DBMS_DATAPUMP.SET_PARAMETER(hdnl,'TABLE_EXISTS_ACTION','SKIP');
This tells the import package what to do if it finds the table already exists during the import, it accepts the following parameters:
SKIP --> Don't import anything on the already exist table.
TRUNCATE --> Truncate the already exist table and import the data.
APPEND --> Leave the currently exist data intact and load the data from the source next to them.
In case you are used wrong parameters or bad combination e.g. using METADATA_FILTER instead of METDATA_REMAP when you are importing to a schema with a different name, you will get a bunch of errors similar to the below cute unclear ones:
ORA-31627: API call succeeded but more information is available
ORA-06512: at "SYS.DBMS_DATAPUMP", line 7143
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79
ORA-06512: at "SYS.DBMS_DATAPUMP", line 4932
ORA-06512: at "SYS.DBMS_DATAPUMP", line 7137
ORA-06512: at line 7
select object_type,count(*) from dba_objects where owner='EPS' group by object_type;
Check the invalid objects:
col object_name for a45
select object_name,object_type,status from dba_objects where owner='EPS' and status<>'VALID';
Step5: [Optional] Delete the dump file from the Target [RDS instance]:
exec utl_file.fremove('DATA_PUMP_DIR','EXPORT_eps_owner_STG_04-03-19.dmp');Check the exist files under DATA_PUMP_DIRdirectory:
select * from table(RDSADMIN.RDS_FILE_UTIL.LISTDIR('DATA_PUMP_DIR')) order by mtime;
References:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html
S3 Bucket creation:
https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
DBMS_DATAPUMP:
https://docs.oracle.com/database/121/ARPLS/d_datpmp.htm#ARPLS356
RDS Master Admin User:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.MasterAccounts.html