Thursday, May 29, 2008

Process Execution( Programming)

What happens when a process is executed.
1. The Kernel will load the required program ( a ELF Object) into memory and also load the runtime linker ( ld.so.1(1)) into memory.
2. The kernel then transfers control initially to the runtime linker.
3. Its runtime linkers job to examine the program loaded and find any dependencies it has ( in the form of a shared objects), load those shared objects into memory , and then bind all of the symbol binds ( function calls, data references, etc...) from the program to each of those dependencies. Of course , as it loads each shared object it must in turn do the same examination on each of them and load any dependencies they require.
4. Once all of the dependencies are loaded and their symbols have been found - the runtime linker will fire the .init sections for each shared object loaded and finally transfer control to the executable, which call main().

What is interposition ( Programming)

Suppose a process is made up of several shared objects, and two shared objects, libX.so and libY.so, export the same symble xy(). under the traditional symbol search model any references to the symbol xy() will be bould to the first instance xy() that is found. So , if libX.so is loaded before libY.so then the instance of xy() within libX.so is used to satisfy all references. The instace of xy() within libX.so is said to interpose on the instance in libY.so

Notes on setting up Cygwin

Get the "setup.exe" tool from www.cygwin.com.

*Setup will lead you through a series of choices (keep the default choices in general for directory names and to install from the internet, but you'll need to pick a local mirror site for the files), and
will let you choose the packages you want to install with a nice GUI.

What I installed

  • I wanted TeX and program development environments, and like to work in X-windows. From the GUI that runs from the setup.exe program (it takes a while for setup to scan the archive the first time) I selected the following files beyond the defaults in addition to the files set by the default settings in each category (setup is smart: selecting something will select most but sometimes not all other necessary files, but the setup program checks for links at the end)
    • Admin
      • cron
    • Devel
      • cvs
      • gcc-g77
      • make
      • (bison, byak, and flex)
    • Editors
      • emacs
      • emacs-X11
      • vim
    • Graphics
      • ghostscript-x11
      • gv
    • Interpreters
      • (gawk; sed is in the Base default)
    • Net
      • openssh
      • rsync
    • Publishing
      • tetex (then don't forget to run texconfig to select paper size and other default options)
      • tetex-base
      • tetex-extras (for BibTeX stuff in my case; run texconfig rehash so TeX knows where to find the extra files)
      • tetex-x11
    • Shells
      • tcsh
    • Text
      • aspell
      • enscript
      • more
    • X11 (no subdirectories, select install for the whole package; the default option isn't quite enough for what I'm doing)
Cygwin's package search page, www.cygwin.com/packages/, is very useful finding missing files or commands. The setup program is smart and automatically gets most of the dependent files, but when you're looking for something you want or are missing...

Path work

Add or set the Windows environment variable HOME to c:\ -- this is where you wind up when you type cd at a Cygwin prompt.
  • The Cygwin initial paths and other setup parameters are in c:\cygwin\etc\profile. After a little editing, my Cygwin path is something like: PATH="/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:.:$PATH" In previous versions /usr/X11R6/bin had to be first in the list to pick up the version of ghostscript (gs) that works in X-Windows, but this seems to have changed in recent releases. The problem before was that a version that didn't have an x11 device was ahead of the X version in the default path. I also added the reference to the current directory at the end of the Cygwin native path but before the Windows path. A full copy of my profile file is below and can be used as a model
Starting the X-Windows manager
refer http://www.astro.umd.edu/~harris/cygwin/index.html

Initialization files

For an XP installation, here are my personalized versions of:
  • profile (modified version of c:\cygwin\etc\profile, written over original version, now saved as profile.org)
  • .xinitrc (modified version of c:\cygwin\etc\X11\xinit\xinitrc saved as c:\.initrc)
  • .cshrc (modified version of c:\cygwin\etc\csh.cshrc saved as c:\.cshrc)
  • .login (modified version of c:\cygwin\etc\csh.login saved as c:\.login
  • .emacs (an emacs initialization file that works for me -- there are many other examples and discussions on the web)
  • startxwin (modified version of c:\cygwin\usr\X11R6\bin\startxwin.sh saved as c:\cygwin\usr\X11R6\startxwin)

Notes on rsync

To rsync to another disk, for instance disk e: (easy to see which are available with the df command):
rsync -avu --delete sourcedir/ /cygwin/e/destdir/
This command removes files that have been removed from the source directory but will not overwrite newer files in the destination,

To rsync to another system with ssh over the net:
rsync -avu --delete -e ssh sourcedir/ username@machine:~/destdir/

To avoid typing passwords for each network transfer:
  • Generate key for ssh with ssh-keygen. Take all defaults including a blank passphrase (otherwise you'll want a passphrase and to invoke an ssh agent, a good idea if you have any security concerns past the most basic ones). Keep track of the file locations.
  • Copy the generated file, id_rsa.pub, to the ~/.ssh directory on the remote machine. Rename it or append it to a file titled authorized_keys. The file must be read-write for the owner only (chmod 600).
  • It is possible to edit this file to restrict access to this mode following instructions in this link.
  • The counterpart to the public file is id_rsa; that may be copied (securely!) to other local machines so you can log in from them as well.

Notes on rsync

To rsync to another disk, for instance disk e: (easy to see which are available with the df command):
rsync -avu --delete sourcedir/ /cygwin/e/destdir/
This command removes files that have been removed from the source directory but will not overwrite newer files in the destination,

To rsync to another system with ssh over the net:
rsync -avu --delete -e ssh sourcedir/ username@machine:~/destdir/

To avoid typing passwords for each network transfer:
  • Generate key for ssh with ssh-keygen. Take all defaults including a blank passphrase (otherwise you'll want a passphrase and to invoke an ssh agent, a good idea if you have any security concerns past the most basic ones). Keep track of the file locations.
  • Copy the generated file, id_rsa.pub, to the ~/.ssh directory on the remote machine. Rename it or append it to a file titled authorized_keys. The file must be read-write for the owner only (chmod 600).
  • It is possible to edit this file to restrict access to this mode following instructions in this link.
  • The counterpart to the public file is id_rsa; that may be copied (securely!) to other local machines so you can log in from them as well.

Grep - global/regular expression/print

Grep has several options, some of those.
-l For listing the names of matching files.
-r Search the directories recursively.
-e If the pattern has leading '-' character.
-w If you want to search whole word, not a part of word.
-C [2-8] prints the two lines of context around each matching line.
/dev/null Appending at the end, forces grep to print the name of the file.
-a or --binary-files=text Forces grep to output the lines even from the files that appear to be binary files.
-I or --binary-files=without-match For eliminating binary file matches.
-lv Lists the names of all files containing one or more lines that do not match.
-L or --files-without-match To list the names of all files that contain no matching lines.

fgrp stands for Fixed grep,egrep is for Extended grep
There are four major variants of grep, controlled by the following options.
`-G'
`--basic-regexp'
Interpret the pattern as a basic regular expression. This is the default.

`-E'
`--extended-regexp'
Interpret the pattern as an extended regular expression.

`-F'
`--fixed-strings'
Interpret the pattern as a list of fixed strings, separated by newlines, any of which is to be matched.

`-P'
`--perl-regexp'
Interpret the pattern as a Perl regular expression.
Examples:

  1. How can I list just the names of matching files?


    grep -l 'main' *.c

    lists the names of all C files in the current directory whose contents mention `main'.

  2. How do I search directories recursively?


    grep -r 'hello' /home/gigi

    searches for `hello' in all files under the directory `/home/gigi'. For more control of which files are searched, use find, grep and xargs. For example, the following command searches only C files:


    find /home/gigi -name '*.c' -print | xargs grep 'hello' /dev/null

    This differs from the command:


    grep -r 'hello' *.c

    which merely looks for `hello' in all files in the current directory whose names end in `.c'. Here the `-r' is probably unnecessary, as recursion occurs only in the unlikely event that one of `.c' files is a directory.

  3. What if a pattern has a leading `-'?


    grep -e '--cut here--' *

    searches for all lines matching `--cut here--'. Without `-e', grep would attempt to parse `--cut here--' as a list of options.

  4. Suppose I want to search for a whole word, not a part of a word?


    grep -w 'hello' *

    searches only for instances of `hello' that are entire words; it does not match `Othello'. For more control, use `\<' and `\>' to match the start and end of words. For example:


    grep 'hello\>' *

    searches only for words ending in `hello', so it matches the word `Othello'.

  5. How do I output context around the matching lines?


    grep -C 2 'hello' *

    prints two lines of context around each matching line.

  6. How do I force grep to print the name of the file?

    Append `/dev/null':


    grep 'eli' /etc/passwd /dev/null

    gets you:


    /etc/passwd:eli:DNGUTF58.IMe.:98:11:Eli Smith:/home/do/eli:/bin/bash

  7. Why do people use strange regular expressions on ps output?


    ps -ef | grep '[c]ron'

    If the pattern had been written without the square brackets, it would have matched not only the ps output line for cron, but also the ps output line for grep. Note that some platforms ps limit the ouput to the width of the screen, grep does not have any limit on the length of a line except the available memory.

  8. Why does grep report "Binary file matches"?

    If grep listed all matching "lines" from a binary file, it would probably generate output that is not useful, and it might even muck up your display. So GNU grep suppresses output from files that appear to be binary files. To force GNU grep to output lines even from files that appear to be binary, use the `-a' or `--binary-files=text' option. To eliminate the "Binary file matches" messages, use the `-I' or `--binary-files=without-match' option.

  9. Why doesn't `grep -lv' print nonmatching file names?

    `grep -lv' lists the names of all files containing one or more lines that do not match. To list the names of all files that contain no matching lines, use the `-L' or `--files-without-match' option.

  10. I can do OR with `|', but what about AND?


    grep 'paul' /etc/motd | grep 'franc,ois'

    finds all lines that contain both `paul' and `franc,ois'.

  11. How can I search in both standard input and in files?

    Use the special file name `-':


    cat /etc/passwd | grep 'alain' - /etc/motd

  12. How to express palindromes in a regular expression?

    It can be done by using the back referecences, for example a palindrome of 4 chararcters can be written in BRE.


    grep -w -e '\(.\)\(.\).\2\1' file

    It matches the word "radar" or "civic".

    Guglielmo Bondioni proposed a single RE that finds all the palindromes up to 19 characters long.


    egrep -e '^(.?)(.?)(.?)(.?)(.?)(.?)(.?)(.?)(.?).?\9\8\7\6\5\4\3\2\1$' file

    Note this is done by using GNU ERE extensions, it might not be portable on other greps.

  13. Why are my expressions whith the vertical bar fail?


    /bin/echo "ba" | egrep '(a)\1|(b)\1'

    The first alternate branch fails then the first group was not in the match this will make the second alternate branch fails. For example, "aaba" will match, the first group participate in the match and can be reuse in the second branch.

Monday, May 19, 2008

Unix useful Commands and traces

*You also can use the " free -m" to see the memory status in megabytes. Change the 'm' to 'k' or 'g' to see it in kilobytes and gigabytes respectively.
*You can use the "watch free" command to see the memory usage in real time. But the display in only in KBs.
* Use the command "fdisk -l" to list all drives you have. This will list the USB devices you have connected as well. It gives the details of partiotions on each disk.

**Try running every command u learn using 'strace' eg 'strace ls' - that would show you the system calls that a command makes and would give you more insights into the working of system/kernel. it would be helpful if you u wish to take system programming.
similarly 'ltrace ' would show you library calls a program makes.

i would suggest to use the "lshw" command, this will give all the info about all the h/w present on your computer