Sunday, June 29, 2008

Bulk replace file extension in Linux

Recently need to do a mass rename of files extension to another in Linux environment. I rename them one by one using "mv" command, which is super slow if talking about 200 over files. Thanks to Sunny who provide me the following solution:

Replace all '.txt' file extension to '.log' extension and display "rename" command in console:
ls *.txt |awk ' {s=substr($1, 1, length($1) - 4); print "mv " s ".txt " s ".log"}'

Executable mode of bulk replace file extension from '.txt' to '.log':
ls *.txt |awk ' {s=substr($0, 1, length($0) - 4); system("mv " s ".txt " s ".log")}'

This save me lots of time, probably yours too. :)

Thanks again, Sunny!

Wicket training over the weekend in UK

Never thought would have a chance to attend any training while in an oversea trip. First time ever to travel 6500 miles, and had the opportunity to take part in a JWeekend Wicket 2 days course. The course was held by Al Maw and Cemal on Fri & Sat, 21st to 22nd March in London.

Normally when I attend trainings back in Malaysia, I experienced lots of "theory" poet reader, which nicely shutdown people's attention, or "here's-the-point-and-i'll-read-it-to-you" type of training. This time, I had lots of practical in the training which firmly sets the experience of using wicket in real. This really helps a lot for an entry level user like me.

The training pace is just nice for me to catch up though on the second day find it a little difficult to digest after the nice Thai lunch! Thanks to Al and Cemal's patience, slowly guide me through all the exercises, make me excited to continue explore the wicket feature.

The training breakdown into modules which easy for me to know what are the topics to be covered. To understand all in a day or 2 is definitely not easy, however, again, emphasized on the experience that Al had, able to explain something in different angle should he finds me not able to understand it. Thanks to his patience to guide me through until I get it!

I'm really grateful to Cemal as well who's willing to setup a laptop for my hand's on lab session. I couldn't bring mine due to my shoulder's injury and was a last minute decision which I did not inform JWeekend prior to the training. This really shows their commitment to take care of their students. So, thanks again, Cemal!

All in all, I enjoyed the training, moreover, I've gained the knowledge that could help me get started in using wicket. Definitely something that you might want to go for if you're looking for a wicket training.

Create/Drop DB in Oracle

Create DB

1) Create file init[dbname].ora to the directory /u01/app/oracle/xxx/xxx/dbs folder (change db_name=[dbname])

2) Create directory /u01/app/oracle/admin/[dbname]

3) Create the directories bdump, cdump and udump in the /u01/app/oracle/admin/[dbname] directory

4) Do a chmod 777 on the /u01/app/oracle/admin/[dbname] directory and sub directories.

5) Prepared the createdb.sql script. Sample script as follow:

CREATE DATABASE dbx
USER SYS IDENTIFIED BY sys
USER SYSTEM IDENTIFIED BY system
LOGFILE GROUP 1 ('/u01/app/oracle/oradata/dbx/redo01.log') SIZE 100M,
GROUP 2 ('/u01/app/oracle/oradata/dbx/redo02.log') SIZE 100M,
GROUP 3 ('/u01/app/oracle/oradata/dbx/redo03.log') SIZE 100M
MAXLOGFILES 5
MAXLOGMEMBERS 5
MAXLOGHISTORY 1
MAXDATAFILES 100
MAXINSTANCES 1
CHARACTER SET WE8ISO8859P15
DATAFILE '/u01/app/oracle/oradata/dbx/system01.dbf' SIZE 325M REUSE
EXTENT MANAGEMENT LOCAL
SYSAUX DATAFILE '/u01/app/oracle/oradata/dbx/sysaux01.dbf' SIZE 325M REUSE
DEFAULT TEMPORARY TABLESPACE tempts1 TEMPFILE '/u01/app/oracle/oradata/dbx/temp01.dbf' SIZE 20M REUSE
UNDO TABLESPACE undotbs01 DATAFILE '/u01/app/oracle/oradata/dbx/undotbs01.dbf' SIZE 200M;

6) Make these files executable (chmod 755 createdb.sql)

7) Switch to user oracle su – oracle

8) Type export ORACLE_SID=[dbname]

9) Type sqlplus /nolog

10) Connect as sysdba

11) get SQL>Connected to an idle instance.

12) Type startup pfile=init[dbname].ora nomount

Should get ORACLE instance started with output as following:

Total System Global Area 707335948 bytes
Fixed Size 452364 bytes
Variable Size 134217728 bytes
Database Buffers 570425344 bytes
Redo Buffers 2240512 bytes

13) Execute the script @/oracle/createdb.sql

14) Run @/u01/app/oracle/product/xxx/rdbms/admin/catalog.sql

Before run the following script, make sure to check the SYSTEM tablespace size whether it is sufficient or not. If not, add datafiles to it, alter the datafile to be auto extend, to resize the tablespace.

15) Run @/u01/app/oracle/product/xxx/rdbms/admin/catproc.sql

After 15) step, check if there's any invalid objects with the following sql (should returns no rows):
SELECT owner, object_name, object_type FROM dba_objects WHERE status='INVALID' ORDER BY owner, object_type, object_name;

16) Connect as user System (sqlplus system/system_password)

17) Run @/u01/app/oracle/product/xxx/sqlplus/admin/pupbld.sql

18) Create Tablespace needed for the database

19) Create User + grantprovide necessary privilege


Drop DB

$ sqlplus /nolog

SQL*Plus: Release 10.1.0.3.0 - Production on Fri Feb 18 20:15:26 2005

Copyright (c) 1982, 2004, Oracle. All rights reserved.

SQL> connect / as sysdba
Connected to an idle instance.

SQL> shutdown abort;
ORACLE instance shut down.

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 612368384 bytes
Fixed Size 790352 bytes
Variable Size 174321840 bytes
Database Buffers 436207616 bytes
Redo Buffers 1048576 bytes

SQL> alter database mount exclusive;

Database altered.

SQL> alter system enable restricted session;

System altered.

SQL> drop database;

Database dropped.

Oracle EE Installation Part II

Let's continue with the second and third section of the Oracle EE Installation.

After all the pre-installation preparation, you can start install Oracle EE by issuing the following command:

$ /directory_path/runInstaller [-silent] [-noconfig] -responseFile responsefilename

For example,
./runInstaller -silent -responseFile /home/oracle/source/database/response/enterprise.rsp

You should be able to see the similar result during the installation:

oracle@linux:~/source/database> ./runInstaller -silent -responseFile /home/oracle/source/database/response/enterprise.rsp
Starting Oracle Universal Installer...
Checking installer requirements...

Checking operating system version:
must be redhat-3, SuSE-9, redhat-4, UnitedLinux-1.0, asianux-1 or asianux-2 Passed
All installer requirements met.

Checking Temp space: must be greater than 80 MB. Actual 337313 MB Passed
Checking swap space: must be greater than 150 MB. Actual 1027 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2007-01-16_11-09-35AM. Please wait ...oracle@raja:~/source/database> Oracle Universal Installer, Version 10.2.0.1.0 Production
Copyright (C) 1999, 2005, Oracle. All rights reserved.

You can find a log of this install session at:
/u01/app/oracle/oraInventory/logs/installActions2007-01-16_11-09-35AM.log
.................................................................................................... 100% Done.
Loading Product Information
................................................................................................................... 100% Done.
Analyzing dependencies
.........................................................................
Starting execution of Prerequisites...
Total No of checks: 11
Performing check for CertifiedVersions
Checking operating system requirements ...
Expected result: One of redhat-3,redhat-4,SuSE-9,asianux-1,asianux-2
Actual Result: SuSE-9
Check complete. The overall result of this check is: Passed
Check complete: Passed
=======================================================================
Performing check for Packages
Checking operating system package requirements ...
Checking for make-3.79; found make-3.80-184.1. Passed
Checking for binutils-2.14; found binutils-2.15.90.0.1.1-32.5. Passed
Checking for gcc-3.2; found Not found. Failed <<<< semmsl="250;" semmsl="250." semmns="32000;" semmns="32000." semopm="100;" semopm="100." semmni="128;" semmni="128." shmmax="536870912;" shmmax="536870912." shmmni="4096;" shmmni="4096." shmall="2097152;" shmall="2097152." max="65536;" max="209689." version="2.6.5-7.97;" version="2.6.5-7.97-smp." ip_local_port_range="1024" ip_local_port_range="1024" rmem_default="262144;" rmem_default="262144." rmem_max="262144;" rmem_max="262144." wmem_default="262144;" wmem_default="262144." wmem_max="262144;" wmem_max="262144." passed ==================================================== atleast="2.3.3-98.28"
passed ====================================================
passed ====================================================
passed ====================================================
passed ====================================================
passed ====================================================
passed ====================================================

Post Installation
After the installation, below are the steps you need to do for post installation:

Make sure ORACLE_HOME environment variable is set

Run the following configuration scripts:
a) Execute as root, /u01/app/oracle/product/xxx/xxx/root.sh

b) Goto sqlplus and login as SYSTEM
# sqlplus /nolog
SQL>connect as sysdba
SQL>system/system
SQL>@/u01/app/oracle/product/10.2.0/rdbms/admin/utlrp.sql

c) At command prompt, run the following script
$ $ORACLE_HOME/bin/genclntst

Phew.. that's all for Silent Mode Oracle EE Installation. There are other places to look into as well, but will save it for other days. ^_^

Oracle EE Installation Part I

Man, compared to previous try to install Oracle XE, Oracle EE (Enterprise Edition) is MUCH MUCH more difficult to configure and to install. Due to the installation is perform from a Windows machine to a Linux server, "ssh -X" command cannot be used, hence, have to adopt manual installation. It took me 3 days to test and find out what minimum steps are needed, and took me another 3 days to install it and create 2 new databases. Most of the time is resource digging on the configuration part and I have forgotten how many installation guides I have gone through. It is just too much information around.

Let me share how it is installed.

Basically to install Oracle EE involve 3 sections:
1) Pre-installation
2) Installation
3) Post-installation

Today, I'm going to share the first section, Pre-Installation.

Before I share the steps, this installation is performed on a Suse 9 Linux platform, with Oracle Database 10g Release 2 (10.2.0.1.0) for Linux x86 version. You can download the installer from here.

Now, again, due to I can't use 'ssh -X' command on a Windows machine, I'll have to go for the Silent Mode installation. (Now I know why to install Oracle through a GUI app is much much recommended!! Silent Mode installation is no fun at all, be warned!) After you've downloaded the installer, unzip it and you should be able to see the Oracle Universal Installer named 'runInstaller' and you'll be touching the 'response' folder alot as well.

This is what you need to prepare the environment for the pre-installation:
1) Check hardware requirement. ie. At least have 1024MB of physical RAM, swapspace, etc. Best to reference the official installation guide on this. You can get the installation guide from here.

2) Check software requirement. For example packages needed prior to install Oracle. For Suse 9, these are needed:
binutils-2.15.90.0.1.1-32.5
gcc-3.3.3-43.24
gcc-c++-3.3.3-43.24
glibc-2.3.3-98.28
gnome-libs-1.4.1.7-671.1
libstdc++-3.3.3-43.24
libstdc++-devel-3.3.3-43.24
make-3.80-184.1
pdksh-5.2.14-780.1
sysstat-5.0.1-35.1
xscreensaver-4.16-2.6

3) Creating Required Operating System Groups and Users (skip this step if already created user and groups)
a) Create user 'oracle'
b) Create group 'dba', 'oper', and 'oinstall'
c) Make 'oracle' user has primary group of 'oinstall' and secondary group of 'dba' and 'oper'

You should be able to do this with the following command:
$ useradd -goinstall -Gdba,oper -pxxxx -m oracle
-g flag is to set primary group
-G flag is to set secondary group
-p flag is to set password
-m flag is to create home directory for the new user

4) Configuring Kernel Parameters
This are the kernel settings needed in Suse 9 platform:
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 262144
net.core.wmem_max = 262144

5) Software location
Oracle RDBMS and other products are installed into /u01 and this directory is typically reserved for software only (i.e. no data/control/redolog/etc. files are to be created/put in this directory or its subdirectories). If you have only one disk, you can put the datafiles into /u01/oradata.
The typical structure of /u01 is
/u01/app/oracle - Base ($ORACLE_BASE)
/u01/app/oracle/oraInventory - Oracle Inventory
/u01/app/oracle/product - Oracle Software
/u01/app/oracle/product/10.2.0/db_1 - Oracle10g Home ($ORACLE_HOME)
/u01/app/oracle/admin - Administrative
/u01/app/oracle/admin/TAR - Admin Support Logs
/u01/app/oracle/admin/sid - Admin Subtree for a SID
/u01/app/oracle/doc - Online Docs

The Base is the base of all Oracle software installation (not just database: development and administartion tools, application server, etc. is also installed here). The ORACLE_BASE environment variable's value must be the absolute path of this location; typically /u01/app/oracle. You should create it before installation.

The above is copied from an installation guide (sorry, don't know which one was it, too many guides!!). For me, I created (with user 'oracle'):
/u01/app/oracle
/u01/app/oracle/oradata
/u01/app/oracle/oraInventory

and set the following environment variables in oracle user's profile:

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export LD_LIBRARY_PATH=/u01/app/oracle/product/10.2.0/db_1/lib
export PATH=$PATH:$ORACLE_HOME/bin

6) Create oraInst.loc file

If you plan to install Oracle products using Oracle Universal Installer in silent or suppressed mode, you must manually create the oraInst.loc file if it does not already exist. This file specifies the location of the Oracle Inventory directory where Oracle Universal Installer creates the inventory of Oracle products installed on the system.

Note:
If Oracle software has been installed previously on the system, the oraInst.loc file might already exist. If the file does exist, you do not need to create a file.

To create the oraInst.loc file, follow these steps:

1. Switch user to root:
$ su - root

2. Change directory as follows:
# cd /etc

3. Use a text editor to create the oraInst.loc file, containing the following lines:
inventory_loc=/u01/app/oracle/oraInventory
inst_group=oinstall

4. Enter the following commands to set the appropriate owner, group, and permissions on the oraInst.loc file:
# chown oracle:oinstall oraInst.loc
# chmod 664 oraInst.loc

7) Create response file
Go to the installer folder search for the 'response' directory, locate a file named 'enterprise.rsp' and edit it. (I suggest to backup a copy first before you edit it)

Unfortunately I'm not familiar with all the settings inside, below is what I find it workable towards the installation (stuck here for quite awhile and make lot of mistakes with multiple combinations configuration setup!) I'll only share a few errors I made or needed attention:

1. Best to comment out whatever you think is not necessary for the installation
2. Must enclose all string value with quotes - ""
3. Put a complete path to 'FROM_LOCATION' field for the products.xml location
4. Take note that SYSMAN and DBSNMP password do not allow to be the same as the username
5. Due to many combination setup inside, I choose the most easiest one, which is:

a) INSTALL_TYPE = "EE"
Installation type of the component

The following choices are available. The value should contain only one of these choices.
EE : Enterprise Edition
SE : Standard Edition
Custom : Custom

b) n_dbType = 1
Determines the type of database to create

This entry should be specified as a number. The valid values that you can use:
1 - General Purpose Starter Database
2 - Transaction Processing Starter Database
3 - Data Warehouse Starter Database
4 - Advanced Configuration

c) n_configurationOption = 1
Determines the type of configuration to perform for the session

This entry should be specified as an number. The valid values that you can use:
1 - Create a Database
2 - Configure an ASM instance
3 - Install Software Only

d) n_performUpgrade = 0
Determines whether to perform an upgrade during the installation or not

If n_performUpgrade=1, then an upgrade will be performed at the end of the installation.
If n_performUpgrade=0, then an upgrade will not be performed

e) n_dbStorageType = 1
Determines the type of storage to use for the the database

This entry should be specified as an number. The valid values that you can use:
1 - Place the data files on a file system
2 - Use Automatic Storage Management (ASM)
3 - Place the datafiles on raw partitions

f) b_enableAutoBackup = false
Determines whether to enable automated backups or not

Pre-requsites for setting this variable:
n_configurationOption=1
n_performUpgrade=0
n_dbType = 1, 2, or 3

g) b_useSamePassword = false
Determines whether the same password is set for each database schema or whether different passwords should be specified.

If b_useSamePassword=true, then you must specify the password that will be used for all accounts using:
- s_superAdminSamePasswd
- s_superAdminSamePasswdAgain

If b_useSamePassword=false, then you must specify the passwords the will be used for each account using:
- sl_superAdminPasswds
- sl_superAdminPasswdsAgain

Please note that this is just a small amount of things that I share it here, in the response file, there're lots more to look into.

After edit this file, you'll need to edit the dbca.rsp file as well. (If you want to create a new DB after the Oracle installation) In dbca.rsp file, is quite straight forward. You only need to look at the 'createDatabase' section and fill in the values accordingly. It is similar to other response file structure where have bunch of key=value pairs.

After all these, you should be ready for the second installation section, which is the real installation part. Will post it as Part II in the blog soon.

Happy installing!

Install Oracle DB Express Edition (XE)

WARNING: this post is for geeks ^_^

Recently I have the chance to install Oracle DB XE 10gR2 version to try out what it is like. I used to be very skeptical on Oracle DB installation because just to create a database already took me like 10 hours or so, I just don't have the courage to think about how long it'll take me to install the Oracle DB server.

To my surprise, this XE version is very fast and easy. This is how I do it to install XE server in Linux Suse 32-bit machine. Oops, just before I list down the steps, this is something you might want to know about XE:
  • Express Edition is limited to a single instance on any server.
  • Express Edition DB instance works with a single database only.
  • Express Edition may be installed on a multiple CPU server, but may only be executed on one processor in any server.
  • The maximum database size in Oracle Database XE is 5 gigabytes (GB). This includes between 0.5 and 0.9 GB for the data dictionary, internal schemas, and temporary space, which leaves just over 4.0 GB for user data.
  • Express Edition may use up to 1 GB RAM of available memory.
XE server installation steps are as follow:
1) Download the installer from Oracle site (I downloaded the rpm version):
http://www.oracle.com/technology/software/products/database/xe/htdocs/102xelinsoft.html

2) Create user 'oracle' if the system has yet to have it
3) Create group 'dba', 'oper', and 'oinstall'
4) Make 'oracle' user has primary group of 'oinstall' and secondary group of 'dba' and 'oper'
You should be able to do this with the following command:
$ useradd -goinstall -Gdba,oper -pxxxx -m oracle
-g flag is to set primary group
-G flag is to set secondary group
-p flag is to set password
-m flag is to create home directory for the new user

5) su to oracle user and run the rpm executable:
$ rpm -ivh oracle-xe-10.2.0.1-1.0.i386.rpm

6) Run the following command
$ /etc/init.d/oracle-xe configure

7) Enter the following configuration information:
  • A valid HTTP port for XE GUI (default is 8080) - web based
  • A valid port for Oracle database listener (default is 1521)
  • A password for SYS and SYSTEM administrative users accounts
  • Whether you want the database to start automatically when the system boots (I choose 'Y' = Yes)
You can always reconfigure the above settings by running the following command:
$ /etc/init.d/oracle-xe configure

To start database manually, run this command:
$ /etc/init.d/oracle-xe start

To stop database manually, run this command:
$ /etc/init.d/oracle-xe stop

8) Go to /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/bin folder and run
$ ./oracle_env.sh

9) Edit .bashrc file in oracle's home directory to include the following:
. /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/bin/oracle_env.sh

10) This is depends if you want to do this. I do the step no 9) in '/etc/profile' file instead. So, whoever that logs in will have the environment variables setup automatically.

Now, depends if you're installing the XE in local machine which you have KDE or any other GUI desktop, you'll be able to access the administration page with this link:
http://127.0.0.1:1521/apex

If you're installing it on a remote server, you need to go into the sqlplus to change a remote HTTP access setting before you can access the web page.

This is how to do it:
1) Start SQL *Plus in the remote server and log in as SYSTEM

2) After login, enter the following command:
SQL> EXEC DBMS_XDB.SETLISTENERLOCALACCESS(FALSE);

After that, you should be able to access the web page, log in as SYSTEM and you should be able to start playing with XE features with a database named 'XE' created for you automatically.

This is how easy it is to setup. For the rest of the installation guide and information, you can get it from here.

Have fun!