Using “tee” to write to files and the terminal May 17, 2007
Posted by Carthik in administration, commands, snippets, ubuntu.37 comments
The utility “tee” is very useful for plumbing on the command line. Curiously enough, it gets its name from the T-splitter used in plumbing, shown below:
Say you want to run a command, and be able to see the output and errors on the screen, and be able to save them to a file. That’s where tee comes in, so you could do a:
$sudo apt-get upgrade 2>&1 | tee ~/apt-get.log
…to run apt-get upgrade and save the output and errors to the file apt-get.log in your home directory.
Purists please excuse the following explanation 🙂 The “2” refers to the “tap” from which the errors pour out (called stderr). The “1” refers to the tap from which the output pours out. The 2>&1 makes the errors to also pour out of the output tap. So then stderr goes to stdout. The pipe “|” redirects the output to tee. Now tee splits the output of the previous command two ways, and puts it both in ~/apt-get.log and in the standard output, which happens to be your screen/terminal.
tee is also handy when you have a small permissions problem. Say you want to write some text to a file “filename.txt” owned by the “root” user – you would just use something like:
$sudo vim filename.txt
and then change the file, right?
Now suppose you want to echo what you write, and write the file, all in one command, you then can use tee thusly:
$echo "localhost 127.0.0.1" | sudo tee filename.txt > /dev/null
This would write the text “localhost 127.0.0.1” to the file filename.txt which is not owned by you. The output of tee itself will go to /dev/null (nothingness) instead of the standard output, which is your terminal.
Don’t lose sleep over this, but someday it will come handy, and when you can figure out why the “sudo” does not apply after the “>” in your command, remember tee and come back here.
For all your command line redirecting needs, and to learn to wield pipes and tees like nunchakus read this excellent page.
^txt2regex$: Regular Expression Tool To Create Regex from Description April 21, 2007
Posted by Carthik in applications, commands, packages, ubuntu.24 comments
^txt2regex$ is a lifesaver. It helps you create regular expression strings in a step by step process, by describing what your regex pattern should do in English (or your own language). The tool can create RegExes for use with 23 different programs, including sed, vim, mysql, and procmail. When you start the program, it will ask you a series of questions like “1. do you want to start matching at the beginning of lines? or 2. search anywhere?” and “this is followed by…. 1. A specific character…” etc… download it and run it and you will see.
Anyone who has worked with regular expressions for searching and optionally replacing stuff in files knows what a godsend then can be if you get the regex down pat – but they would also know what a time sink they can be if you can’t whip up exactly what you want. In the past, when faced with this kind of a situation, I would read man pages, books, experiment, fail and then, finally, succeed after a good half hour or so. txt2regex is a tool that eliminates the confusion. Totally.
You can install txt2regex on Ubuntu by doing a:
$sudo apt-get install txt2regex
Among its features include the ability to print a list of characteristics of the regular expression syntax for various tools, a history tool which keeps track of you past regexes, and some pre-built regexes that are often used – for dates, times and numbers.
For example,
$txt2regex --all --make number3
will create the regex for all supported tools. The regex will match a number of the form “34,412,069.90”
Here’s the output:
carthik@milan:~$ txt2regex --all --make number3 ### number LEVEL 3: level 2 plus optional commas, like: 34,412,069.90 RegEx awk : [+-]?[0-9]!!(,[0-9]!!)*(\.[0-9]!!)? RegEx ed : [+-]\?[0-9]\{1,3\}\(,[0-9]\{3\}\)*\(\.[0-9]\{2\}\)\? RegEx egrep : [+-]?[0-9]{1,3}(,[0-9]{3})*(\.[0-9]{2})? RegEx emacs : [+-]?[0-9]!!\(,[0-9]!!\)*\(\.[0-9]!!\)? RegEx expect : [+-]?[0-9]!!(,[0-9]!!)*(\.[0-9]!!)? RegEx find : [+-]?[0-9]!!\(,[0-9]!!\)*\(\.[0-9]!!\)? RegEx gawk : [+-]?[0-9]{1,3}(,[0-9]{3})*(\.[0-9]{2})? RegEx grep : [+-]\?[0-9]\{1,3\}\(,[0-9]\{3\}\)*\(\.[0-9]\{2\}\)\? RegEx javascript: [+-]?[0-9]{1,3}(,[0-9]{3})*(\.[0-9]{2})? RegEx lex : [+-]?[0-9]{1,3}(,[0-9]{3})*(\.[0-9]{2})? RegEx lisp : [+-]?[0-9]!!\\(,[0-9]!!\\)*\\(\\.[0-9]!!\\)? RegEx mawk : [+-]?[0-9]!!(,[0-9]!!)*(\.[0-9]!!)? RegEx mysql : [+-]?[0-9]{1,3}(,[0-9]{3})*(\\.[0-9]{2})? RegEx ooo : [+-]?[0-9]{1,3}(,[0-9]{3})*(\.[0-9]{2})? RegEx perl : [+-]?[0-9]{1,3}(,[0-9]{3})*(\.[0-9]{2})? RegEx php : [+-]?[0-9]{1,3}(,[0-9]{3})*(\.[0-9]{2})? RegEx postgres : [+-]?[0-9]{1,3}(,[0-9]{3})*(\\.[0-9]{2})? RegEx procmail : [+-]?[0-9]!!(,[0-9]!!)*(\.[0-9]!!)? RegEx python : [+-]?[0-9]{1,3}(,[0-9]{3})*(\.[0-9]{2})? RegEx sed : [+-]\?[0-9]\{1,3\}\(,[0-9]\{3\}\)*\(\.[0-9]\{2\}\)\? RegEx tcl : [+-]?[0-9]!!(,[0-9]!!)*(\.[0-9]!!)? RegEx vbscript : [+-]?[0-9]{1,3}(,[0-9]{3})*(\.[0-9]{2})? RegEx vi : [+-]\{0,1\}[0-9]\{1,3\}\(,[0-9]\{3\}\)*\(\.[0-9]\{2\}\)\{0,1\} RegEx vim : [+-]\=[0-9]\{1,3}\(,[0-9]\{3}\)*\(\.[0-9]\{2}\)\=
$txt2regex --help
prints out a short help message
and
man txt2regex
gives some more info.
What would be handy is if txt2regex had an extension that allowed one to deconstruct a regex – give it a regex and it tells you what it does in plain English. Also, I cannot seem to create regexes for the mod_rewrite module in apache. I suspect that since mod_rewrite supports POSIX regexes, I could just run with one or the other of the regexes created by txt2regex. Since I haven’t tried it, I can’t say which one of the 23, just yet.
Saving Sound Volume Levels Across Reboots March 20, 2007
Posted by Carthik in commands, ubuntu.15 comments
I have a weird problem with a desktop where the volume settings for PCM, Microphone etc are all set to mute when I restart the machine. I then have to use the alsamixer command line tool to set the volumes to desired levels. This annoyed the hell out of me.
Finally I found a solution to this problem. So, if your computer has a volume memory problem, the following might be useful.
To set the volume and other sound settings to your desired level and save the settings, do the following:
Delete the /etc/asound.conf file if it exists– Read Daniel’s Comment Below- Use the alsamixer command to set the various volume levels etc
- Save the settings by using the command: $sudo alsactl store
And that is it!
Another tip I read on lifehacker: This works to suppress the volume when you boot up a laptop in a place where it is not supposed to annoy others — think library, seminar, conference…. Take an old, non-usable headphone, and snip off the wire just above the jack that plugs in to the speaker output of the computer. Plug this into your laptop before you boot up to prevent annoying people with the startup sounds as the computer boots up! It is a simple solution to a serious problem.
Find Hardware Specs (Details) on your Computer February 18, 2007
Posted by Carthik in commands, snippets, ubuntu.138 comments
I stumbled upon the nifty “lshw” tool today. lshw lists your hardware. Try it now:
$sudo lshw
You can get specific details by using the -C flag:
$sudo lshw -C disk
will list all you hard disks.
It create an html page with your hardware details if you do a:
$sudo lshw -html > your-file-name.html
I generated one for my laptop and put it up for future reference. Now I have an answer to the question, “what network adapter do you have, or what wireless driver are you using?”
I know, this is something probably all of you know already. I just found out about this little tool today. I expect this info might be useful for those of you just learning the ropes, like me.
“True” Word Count In LaTeX February 7, 2007
Posted by Carthik in commands, packages, Readers' Tips.42 comments
By way of Wei comes this little nugget of useful information of the kind I love.
If you were to count the number of words in a LaTeX document using the “wc” command, you will find that you have counted, in addition to the words you wrote, all the LaTex formatting text, like the “\paragraph”s and the “\textit”s.
Of course if you use Kile like I do, all you have to do is go to “File -> Statistics” to see the word count. But if you don’t use Kile, then you can follow Wei’s advice and install and use the “untex” package by doing a:
$sudo apt-get install untex
and then a:
$untex source.tex > target && wc -w target
to count the number of words in the file named “source.tex”.
Alternatively, you can use this online tool to count the words.
A word of caution here — untex does not ignore equations, and so the output of the word count might be off by a bit. If you are a perfectionist, I would recommend using detex instead. There is no seperate package for detex, it ships in the Ubuntu package texlive-extra-utils.
If your document has citations, references, and include other files etc, the only reasonably efficient way to count the words in the final result is to convert the pdf file to text and then to count the words. Here is a command that will help you do that:
$pdftotext file.pdf - | egrep -E '\w\w\w+' | iconv -f ISO-8859-15 -t UTF-8 | wc
pdftotext is a command line utility provided by Xpdf. You may have to tweak the charsets in the previous command.
gnome-open: Open Anything from the Command Line December 16, 2006
Posted by Carthik in commands, gnome, guides, ubuntu.98 comments
There is a command called gnome-open which I find very handy. I thought of sharing it with you, and was searching for documentation on it, and found none, neither on the web using google, nor on the system. No man page entry, or info entry. All the more reason I should document what I know about it.
Simply put, the command
gnome-open
opens the item specified by the url with the preferred GNOME app for that file/mime-type.
In a sense, this command resembles the universal “open” command on Mac OSX.
Read on for examples…
(more…)
SSH Tunnel + SOCKS Proxy Forwarding = Secure Browsing December 8, 2006
Posted by Carthik in applications, commands, guides, servers, ubuntu.221 comments
When you are at the coffee shop, or at a conference, and you are not sure that you want to send all your data over the wi-fi network in plaintext, you want a secure tunnel to browse. This happened to me recently and I stumbled across a neat feature of openssh (the ssh client on everyone’s computer). The wonders of ssh never cease to amaze me!
You can use the “-D” flag of openssh to create a SOCKS proxy.
The command first:
$ssh -D 9999 username@ip-address-of-ssh-server
This of course connects you to the server specified by “ip-address-of-ssh-server”. Needless to say, you (username) must have an ssh account on the server. In addition, this will create a SOCKS proxy on port “9999” of your computer. This is a tunnel to the server. Now all you have to do is set the preference in Firefox to use a SOCKS proxy. The proxy is, of course, “localhost”, with the port 9999.
Now when you browse, all the connections you make to websites will seem to originate from the server to which you SSH-ed. In addition, all outgoing and incoming data for the browsing session will be encrypted since it passes through the SSH connection.
Other applications (like email clients) may also support SOCKS proxies. If any of them, you can look into using proxychains(there’s an Ubuntu package).
You can misuse this technology to circumvent paranoid browsing firewalls, even to pretend you are wherever your ssh server is located – so you can work around country-based blocks etc. I use it for the very unromantic reason that I don’t want some aspiring cracker to sneak up on me when I am in public.
Updates:
- Kees Cook tells us how to tunnel DNS lookups, so snoopy folks can’t even figure out what your are browsing, and the evil ones can’t DNS-phish you
- Don McArthur points out his excellent article that addresses the same issue
- verevi says the FoxyProxy extension will make things easier for you on the Firefox side of things
Thanks a lot for the tips and pointers, folks.
Disable Touchpad Temporarily When Typing September 20, 2006
Posted by Carthik in commands, guides, looks and feel, ubuntu.64 comments
Earlier, I wrote about how to enable/disable your touchpad using the synclient command.
Recently, I faced a different problem. When typing on the new laptop that I got, my thumb often accidentally brushes the touchpad, and this leads to me continuing to type things in entirely a different place. This is because the thumb moves the mouse pointer to a different spot that where the typing cursor is, and then there is an accidental click. Argh! very annoying.
The good news is, I figured out how to fix this using syndaemon!
syndaemon watches activity on the keyboard and can disable your synaptics touchpad for a variable period after it detects activity on the keyboard. Here’s how I use it:
First, I edited the /etc/X11/xorg.conf
file and added the Option “SHMConfig” “on” line to the section called “Input Device” for the Synaptic Touchpad input device.
Then I restarted X (by using the ctrl+alt+backspace key combination).
Once I was logged in, I used syndaemon as $syndaemon -t -d
The -t option specifies that only the tapping and scrolling actions are to be disabled, I can still move the cursor around while typing on the keyboard.
The -d option asks syndaemon to run in the background as a daemon, so I don’t have to keep the terminal open after executing the command.
You can disable the touchpad entirely by not using the -t option.
By default, syndaemon disables the touchpad for 2 seconds after the last keyboard activity. You can change this by specifying the idle-time using the -i option. Read the manual for all details: $man syndaemon.
To make syndaemon start up by default each time you login, add it to the list of Startup Programs in System->Preferences->Sessions. I have the following command added there now: syndaemon -t -d
. Log out and log back in to see if its working for you.
Normalize the Gain (Playback Volume) of your MP3s September 11, 2006
Posted by Carthik in applications, commands, guides, packages, ubuntu.50 comments
I confess to the grave sin of storing part of my music collection in the mp3 format – mostly music I have had around for ages. There, now that that is out of the way, let’s move on to making that collection friendlier.
I have the habit of going to sleep while listening to a little light music. Some of the songs in my collection would sound louder than the others. So when I have a couple of classical music tracks lined up with Paul McCartney and Guns ‘N Roses, I sometimes get jolted awake by the difference in the playback volume. So off I went looking for solutions.
My main concerns when I started my research were:
- My mp3s should not be irretrievably changed into something I end up hating
- I did not want all songs sounding as loud as the loudest song
- The algorithm or method used should be free of the application used for music playback
Then came the choices. In the end it all came down to two apps: Normalize and mp3gain. Now how do I decide which one’s the one for me without learning DSP? Simple, search some more.
Fortunately, besides the websites for the applications, I ran into this thread at jwz’s livejournal. Thank god for geeks-who-start-nightclubs and their friends! I also followed most of the replaygain technical outline, since mp3gain is an implementation of the replaygain idea.
Normalize is simple, perhaps a little too simple.
mp3gain works by figuring out how loud the music actually sounds to the human ear, and then figuring out how much positive or negative gain to apply to the mp3 file to “fix” it. I was happy to note that it does not use the peak volume in a file as a benchmark for normalization. It applies changes by modifying the mp3 file in some fashion, but I read that the change is reversible since mp3gain writes some tags (not ID3 tags) to the mp3s for possible later undoing. This fact also means that one you use mp3gain on a file, if you try doing it again, it takes a lot less time – almost no time. So in the future if I add files to the collection, and I don’t remember what files I added, I can run mp3gain on the entire collection, and it will complete much faster than the first time.
To cut to the chase, I decided to go with mp3gain. I installed the mp3gain package (using $sudo apt-get install mp3gain), and set out to normalize my entire music collection. (For those of you who also have .oggs, check out vorbisgain).
Edit: for an alternative command, with explanation, and a way to use album-gain for songs from albums, visit porges
The following command, executed from the directory where I store my music normalized all the files in my collection:
$find . -type f -iname '*.mp3' -print0 | xargs -0 mp3gain -r -k
What it does is, it finds all files (type -f) in the present directory (.) with a name that ends in “.mp3” and creates a list of the same. The output of this find command is then piped to the mp3gain program. The options that I ended up using for mp3gain, -r and -k dictate that the calculated track gain will be applied automatically to normalize the volume, and that files should be protected against “clipping” by lowering the applied gain if the required gain seems likely to clip the sound. Concoct your own recipe by referring to the man page for mp3gain.
After around 16 hours of processing on my old PIII desktop, my massive 50 GB music collection now agrees on what is an acceptable volume. Needless to say, I sleep in peace.
Installing Packages on Computers with Slow Connections Redux July 8, 2006
Posted by Carthik in administration, commands, guides, maintenance, ubuntu.15 comments
Earlier, I had written about using apt-zip to upgrade computers on slow internet connections by using a faster machine to do the downloads. However, since that involves understanding how apt-zip works, and a small learning curve, here is a hack that should work pretty well, in spite of it’s hackish nature.
The tip here will let you install new packages (and their dependencies) or upgrade a system by using a faster internet connection to do the downloading. First we get the URIs for the pacakges to be downloaded, then we download the packages and transport them to the computer with the slower connection. Thanks to Ewan for this tip posted to the ubuntu-users list a long time ago.
The steps:
1) On the computer you wish to install something new on (or upgrade), do a $sudo apt-get update
2) Then use apt-get to generate a list of the packages it needs to download, in order to install the package that you need, but not download them:
$sudo apt-get -qq --print-uris install name-of-package | cut -d\' -f 2 > urilist
this gives you a file ‘urilist’ in the same directory as the one in which you ran the previous command, with a list of files to download.
3) Take your list to a machine with a fast internet connection and download the packages using wget:
$wget -i < urilist
4) Take your newly downloaded debs home and copy them into the
/var/cache/apt/archives/
directory
5) Rerun the same apt-get command, but without any special parameters:
$ sudo apt-get install name-of-package
apt-get should tell you the quantity of packages that it will install,
and how much it will download. The download amount should be zero since
the packages are already downloaded.