Frequently Asked Questions

  1. How do you handle locking?
  2. How do you create relationships?
  3. How are errors handled?
  4. Is it possible to Roll Back operations?
  5. Can Unicage coexist with my database?
  6. How do I ensure that access to the data is secure?
  7. How do you defend against command insertion attacks?
  8. If you write web applications in CGI, won’t there be too many processes?



1. How do you handle locking?

You use the “ulock” command. The ulock command creates a lock file with exclusive access. It waits until that lock file no longer exists. An example of this is below:

Example

				
				if ulock lockfile; then
					###############################
					# Commands to be executed exclusively
					###############################
					# Finally, delete the lock file
					rm lockfile
				fi
				

The ulock command supports timeouts, cleanup of old files and shared locking.



2. How do you create relationships?

Unicage uses structured text files as the database. In order to create relationships, Unicage has many commands that join together several text files and outputs the result to standard output. You can specify the column number or tag name and perform inner/outer joins.

Example

Join files Master and Tran using the first column of Master and the second column of Tran as the key field.

				
				$ cat Master
				001 Japan
				002 America
				$ cat Tran
				a 001 100
				b 001 200
				c 002 300
				d 003 400
				$ join1 key=2 Master Tran > Result
				$ cat Result
				a 001 Japan 100
				b 001 Japan 200
				c 002 America 300
				



3. How are errors handled?

In shell scripts you can augment the end status code of each command. In bash, the end status code of each command that is joined together with pipes is stored in the array variable PIPESTATUS. If all elements are 0 then all commands finished normally.
We show an example of error handling code below. The “plus” command outputs to standard output the sum of all of the arguments passed to it.

				
				function error_check()
				{
					[ $(plus ${PIPESTATUS[@]}) -eq 0 ] && return 0
					echo error:$1
					exit 1
				}
				command1   |
				command2   |
				command3   > result
				error_check $LINENO
				



4. Is it possible to Roll Back operations?

Unicage commands process an input file and output the results to another file. In other words, in all instances the original data and the result data are kept in separate files, so all of the important files are saved. For example, when updating the master file, as shown below all of the files from before and after the update are preserved. We use symbolic links to refer to updated files with the same name.

Simply by changing the symbolic link you can roll back to any point in time.

				
				if ulock lockfile; then
				time=$(date +%Y%m%d%H%M%S).$$
				upl key=1 Master Tran > Master.$time
				rm Master
				ln -s Master.$time Master
				rm lockfile
				fi
				



5. Can Unicage coexist with my database?

Unicage provides client commands to connect to most databases. For example, for connecting to SQL Server, if you install unixODBC and FreeTDS and set it up properly, then using the command below you can access the database from the shell script.

$ tsql -S mssql -U username -P password

For Oracle databases, install “Instant Client” and connect to the database with the “sqlplus” command.

You can use these commands to export data such as user data that is being managed one-dimensionally in the database for use with Unicage

For transaction data, you can export the data as a CSV and import that into the shell script.



6. How do I ensure that access to the data is secure?

In general when securing data access in Unicage, we create a system where the client accesses the data via a web server running CGI or other server side programming and do not allow the client to log in directly to the servers that are holding the data. Moreover, we implement data access controls at the application level using the Unix privileges system. For important data such as private personal information, we use encryption. When de-encrypting, we use pipes to stream the data directly to commands so the unencrypted data is never propagated to a file or to memory.



7. How do you defend against command insertion attacks?

The input data for Unicage commands is stored in files and is processed as a file so it is rare to set data in a shell variable. Unicage offers commands that llow you to check data while it is still stored in a file. Also, Unicage provides a script that checks your programming to see if you have any “exec/eval/:/source” commands, if you have shell variables at the beginning of lines and to check the system variables for awk. This script eliminates code that could allow unauthorized access.



8. If you write web applications in CGI, won’t there be too many processes?

If there are a few thousands of processes, then a standard server today will have no problem. If there are tens of thousands of processes then you need to use the following “wait your turn” processing in the CGI.

				
				semwait --less_than 10 "semaphore.*"
				touch semaphore.$$
				##############
				# Processing
				##############
				rm semaphore.$$
				

If you have so many concurrent accesses that this method cannot cope, then set up FastCGI to convert the web application to a server.